[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 720136712 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009138] LSM: Security Framework initializing [ 0.010042] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012062] *** VALIDATE selinux *** [ 0.019495] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023726] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027075] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029010] *** VALIDATE tmpfs *** [ 0.030181] *** VALIDATE proc *** [ 0.031234] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033250] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036000] Spectre V2 : User space: Vulnerable [ 0.036008] Speculative Store Bypass: Vulnerable [ 0.039584] debug: unmapping init [mem 0xffffffffaa659000-0xffffffffaa660fff] [ 0.042169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043649] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044021] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.046964] ... value mask: 0000ffffffffffff [ 0.047014] ... max period: 00007fffffffffff [ 0.048010] ... fixed-purpose events: 3 [ 0.049011] ... event mask: 000000070000000f [ 0.050268] rcu: Hierarchical SRCU implementation. [ 0.052412] smp: Bringing up secondary CPUs ... [ 0.053510] x86: Booting SMP configuration: [ 0.054019] .... node #0, CPUs: #1 #2 #3 [ 0.057205] smp: Brought up 1 node, 4 CPUs [ 0.059010] smpboot: Max logical packages: 1 [ 0.060017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.084650] node 0 deferred pages initialised in 22ms [ 0.089094] devtmpfs: initialized [ 0.090479] x86/mm: Memory block size: 128MB [ 0.094670] gcov: version magic: 0x41383552 [ 0.096327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.102077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.104360] pinctrl core: initialized pinctrl subsystem [ 0.107161] [ 0.107681] ************************************************************* [ 0.109011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.112013] ** ** [ 0.114009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.116013] ** ** [ 0.118011] ** This means that this kernel is built to expose internal ** [ 0.119000] ** IOMMU data structures, which may compromise security on ** [ 0.119000] ** your system. ** [ 0.119000] ** ** [ 0.119000] ** If you see this message and you are not debugging the ** [ 0.119000] ** kernel, report this immediately to your vendor! ** [ 0.119000] ** ** [ 0.119000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.119000] ************************************************************* [ 0.119000] NET: Registered protocol family 16 [ 0.119000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.124058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.125000] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.130134] cpuidle: using governor menu [ 0.132000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.134697] PCI: Using configuration type 1 for base access [ 0.137237] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.149124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.150032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.152186] cryptd: max_cpu_qlen set to 1000 [ 0.155925] ACPI: Added _OSI(Module Device) [ 0.157013] ACPI: Added _OSI(Processor Device) [ 0.159013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.161011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.168345] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176179] ACPI: Interpreter enabled [ 0.177049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.178025] ACPI: Using IOAPIC for interrupt routing [ 0.181116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.185491] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.194000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.194000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.194000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.194128] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.200804] acpiphp: Slot [2] registered [ 0.202170] acpiphp: Slot [5] registered [ 0.203086] acpiphp: Slot [6] registered [ 0.205144] acpiphp: Slot [3] registered [ 0.206088] acpiphp: Slot [4] registered [ 0.208133] acpiphp: Slot [7] registered [ 0.209100] acpiphp: Slot [8] registered [ 0.211116] acpiphp: Slot [9] registered [ 0.212170] acpiphp: Slot [10] registered [ 0.214181] acpiphp: Slot [11] registered [ 0.215074] acpiphp: Slot [12] registered [ 0.216092] acpiphp: Slot [13] registered [ 0.218098] acpiphp: Slot [14] registered [ 0.219088] acpiphp: Slot [15] registered [ 0.221120] acpiphp: Slot [16] registered [ 0.223170] acpiphp: Slot [17] registered [ 0.225126] acpiphp: Slot [18] registered [ 0.226159] acpiphp: Slot [19] registered [ 0.228087] acpiphp: Slot [20] registered [ 0.229082] acpiphp: Slot [21] registered [ 0.231096] acpiphp: Slot [22] registered [ 0.232190] acpiphp: Slot [23] registered [ 0.234096] acpiphp: Slot [24] registered [ 0.236126] acpiphp: Slot [25] registered [ 0.237191] acpiphp: Slot [26] registered [ 0.239186] acpiphp: Slot [27] registered [ 0.241138] acpiphp: Slot [28] registered [ 0.243263] acpiphp: Slot [29] registered [ 0.245204] acpiphp: Slot [30] registered [ 0.247357] acpiphp: Slot [31] registered [ 0.250132] PCI host bridge to bus 0000:00 [ 0.252025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256110] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.259022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.267026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.271118] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.274524] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.279807] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287343] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297769] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.304142] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.310017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316879] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.321370] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328009] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334012] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.350017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.356015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362884] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.373016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.382018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431425] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.443016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.452016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.473019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490233] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.492424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.494276] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.496303] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.498293] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.504525] iommu: Default domain type: Passthrough [ 0.505000] SCSI subsystem initialized [ 0.505000] ACPI: bus type USB registered [ 0.506077] usbcore: registered new interface driver usbfs [ 0.508080] usbcore: registered new interface driver hub [ 0.510073] usbcore: registered new device driver usb [ 0.512173] pps_core: LinuxPPS API ver. 1 registered [ 0.514009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.518058] PTP clock support registered [ 0.520131] EDAC MC: Ver: 3.0.0 [ 0.521700] PCI: Using ACPI for IRQ routing [ 0.522703] NetLabel: Initializing [ 0.524011] NetLabel: domain hash size = 128 [ 0.526008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.528178] NetLabel: unlabeled traffic allowed by default [ 0.530268] vgaarb: loaded [ 0.532427] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.534014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.542451] clocksource: Switched to clocksource kvm-clock [ 0.783517] VFS: Disk quotas dquot_6.6.0 [ 0.785523] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.789542] *** VALIDATE ramfs *** [ 0.790869] *** VALIDATE hugetlbfs *** [ 0.792692] pnp: PnP ACPI init [ 0.795579] pnp: PnP ACPI: found 6 devices [ 0.819192] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.823702] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.826508] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.829767] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.832679] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.835443] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.839133] NET: Registered protocol family 2 [ 0.842041] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.848554] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.854214] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.863408] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.869948] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874595] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.879510] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887603] NET: Registered protocol family 1 [ 0.890950] RPC: Registered named UNIX socket transport module. [ 0.893988] RPC: Registered udp transport module. [ 0.896435] RPC: Registered tcp transport module. [ 0.898475] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901600] NET: Registered protocol family 44 [ 0.903505] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.906176] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.908625] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.911232] PCI: CLS 0 bytes, default 64 [ 0.913227] Unpacking initramfs... [ 3.475894] debug: unmapping init [mem 0xffff8bb0bcc64000-0xffff8bb0bffcffff] [ 3.481239] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.483681] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.487180] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.304743] Initialise system trusted keyrings [ 4.306378] Key type blacklist registered [ 4.308244] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.318503] zbud: loaded [ 4.321804] *** VALIDATE nfs *** [ 4.322935] *** VALIDATE nfs4 *** [ 4.324357] pstore: using deflate compression [ 4.328176] Platform Keyring initialized [ 4.591749] NET: Registered protocol family 38 [ 4.594968] Key type asymmetric registered [ 4.596636] Asymmetric key parser 'x509' registered [ 4.598530] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.601572] io scheduler mq-deadline registered [ 4.603525] io scheduler kyber registered [ 4.605144] io scheduler bfq registered [ 4.610983] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.614655] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.617503] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.621671] ACPI: Power Button [PWRF] [ 4.628850] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.636753] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.649934] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.680254] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.711285] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.883760] Non-volatile memory driver v1.3 [ 4.885470] Linux agpgart interface v0.103 [ 4.918482] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 4.920821] vda: detected capacity change from 0 to 74838016 [ 4.956159] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.959728] vdb: detected capacity change from 0 to 1073741824 [ 4.981859] libphy: Fixed MDIO Bus: probed [ 4.994559] usbcore: registered new interface driver usbserial_generic [ 4.996801] usbserial: USB Serial support registered for generic [ 4.999374] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.003715] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.005677] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.008497] mousedev: PS/2 mouse device common for all mice [ 5.012517] rtc_cmos 00:05: RTC can wake from S4 [ 5.017189] rtc_cmos 00:05: registered as rtc0 [ 5.018642] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.020861] intel_pstate: CPU model not supported [ 5.023749] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.031437] hid: raw HID events driver (C) Jiri Kosina [ 5.034623] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.036800] usbcore: registered new interface driver usbhid [ 5.036805] usbhid: USB HID core driver [ 5.036912] drop_monitor: Initializing network drop monitor service [ 5.038121] Initializing XFRM netlink socket [ 5.042531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.055443] NET: Registered protocol family 10 [ 5.058455] Segment Routing with IPv6 [ 5.060049] NET: Registered protocol family 17 [ 5.062403] mpls_gso: MPLS GSO support [ 5.067767] RAS: Correctable Errors collector initialized. [ 5.072081] AVX version of gcm_enc/dec engaged. [ 5.074085] AES CTR mode by8 optimization enabled [ 5.181778] sched_clock: Marking stable (5181694580, 0)->(6436097708, -1254403128) [ 5.192354] registered taskstats version 1 [ 5.194836] Loading compiled-in X.509 certificates [ 5.196723] zswap: loaded using pool lzo/zbud [ 5.240691] Key type big_key registered [ 5.258378] Key type encrypted registered [ 5.259863] ima: No TPM chip found, activating TPM-bypass! [ 5.261914] ima: Allocated hash algorithm: sha1 [ 5.263767] ima: No architecture policies found [ 5.265732] evm: Initialising EVM extended attributes: [ 5.267807] evm: security.selinux [ 5.269646] evm: security.ima [ 5.271576] evm: security.capability [ 5.272752] evm: HMAC attrs: 0x1 [ 5.275897] rtc_cmos 00:05: setting system clock to 2026-08-07 14:43:07 UTC (1786113787) [ 5.285127] debug: unmapping init [mem 0xffffffffab603000-0xffffffffab7fffff] [ 5.289204] debug: unmapping init [mem 0xffffffffaa382000-0xffffffffaa658fff] [ 5.302164] Write protecting the kernel read-only data: 28672k [ 5.308126] debug: unmapping init [mem 0xffffffffa8a03000-0xffffffffa8bfffff] [ 5.311692] debug: unmapping init [mem 0xffffffffa9314000-0xffffffffa93fffff] [ 5.355210] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.371481] systemd[1]: Detected virtualization kvm. [ 5.374711] systemd[1]: Detected architecture x86-64. [ 5.376989] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.411886] systemd[1]: No hostname configured. [ 5.413968] systemd[1]: Set hostname to . [ 5.420260] random: systemd: uninitialized urandom read (16 bytes read) [ 5.422880] systemd[1]: Initializing machine ID from random generator. [ 5.734658] random: systemd: uninitialized urandom read (16 bytes read) [ 5.741468] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.748258] random: systemd: uninitialized urandom read (16 bytes read) [ 5.752780] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 5.771446] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.348449] device-mapper: uevent: version 1.0.3 [ 7.350671] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 8.878093] virtio_net virtio0 ens2: renamed from eth0 [ 8.899781] random: fast init done [ OK ] Started udev Coldplug all Devices. [ 9.302806] scsi host0: ata_piix [ 9.608936] scsi host1: ata_piix [ 9.619452] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.629261] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.349149] random: crng init done [ 15.350798] random: 7 urandom warning(s) missed due to ratelimiting [ 17.794283] dracut-initqueue[566]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 19.778680] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 22.490245] printk: systemd: 26 output lines suppressed due to ratelimiting [ 23.198836] SELinux: Disabled at runtime. [ 23.269719] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 23.279691] systemd[1]: Detected virtualization kvm. [ 23.281572] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 25.096488] systemd[1]: initrd-switch-root.service: Succeeded. [ 25.100569] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 25.114737] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 25.120616] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 25.125351] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 25.133540] systemd[1]: Starting Journal Service... Starting Journal Service... [ 25.148781] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [[ 25.528217] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.876500] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.535622] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.667023] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 28.196434] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.307906] EDAC sbridge: Ver: 1.1.2 [ 30.921458] Key type dns_resolver registered [ 31.268688] NFS: Registering the id_resolver key type [ 31.271681] Key type id_resolver registered [ 31.273479] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg243-client login: [ 72.272287] libcfs: loading out-of-tree module taints kernel. [ 72.357783] Key type ._llcrypt registered [ 72.359209] Key type .llcrypt registered [ 72.578777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 72.589495] alg: No test for adler32 (adler32-zlib) [ 73.641745] Lustre: Lustre: Build Version: 2.17.55_1_gd52575f [ 73.955288] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 75.575338] Key type lgssc registered [ 76.650415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 143.055236] Lustre: Mounted lustre-client [ 146.146863] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 152.816539] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing check_logdir /tmp/testlogs/ [ 155.733368] Lustre: DEBUG MARKER: oleg243-client.virtnet: executing yml_node [ 158.287480] Lustre: DEBUG MARKER: Client: 2.17.55.1 [ 159.881654] Lustre: DEBUG MARKER: MDS: 2.17.55.1 [ 161.346797] Lustre: DEBUG MARKER: OSS: 2.17.55.1 [ 162.187544] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 7 10:45:43 EDT 2026 [ 163.807928] Lustre: lustre-OST0000-osc-ffff8bb118bb1000: disconnect after 20s idle [ 170.677729] Lustre: Unmounted lustre-client [ 182.287419] Key type lgssc unregistered [ 182.477659] LNet: 8355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 182.485679] LNetError: 8355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 182.496809] LNet: Removed LNI 192.168.202.43@tcp [ 182.940399] Key type .llcrypt unregistered [ 182.944825] Key type ._llcrypt unregistered [ 197.104183] Key type ._llcrypt registered [ 197.106903] Key type .llcrypt registered [ 197.217915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 197.229494] alg: No test for adler32 (adler32-zlib) [ 198.090885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 198.134758] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 198.172864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 200.602606] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 200.716312] LNet: 9238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 200.719473] LNetError: 9238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 200.726540] LNet: Removed LNI 192.168.202.43@tcp [ 201.041212] Key type .llcrypt unregistered [ 201.044144] Key type ._llcrypt unregistered [ 204.208324] Key type ._llcrypt registered [ 204.210905] Key type .llcrypt registered [ 204.931853] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:46:26 (1786113986) [ 205.107807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 205.118448] alg: No test for adler32 (adler32-zlib) [ 205.947994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.011704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.023384] LNet: 9647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.038888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.076667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 206.090518] LNet: 9654:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 209.773876] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:46:31 (1786113991) [ 209.850786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.875797] LNet: 10225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 209.905405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.963362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 213.456201] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:46:34 (1786113994) [ 213.501872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.514907] LNet: 10809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 213.532747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.579405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 216.998508] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:46:38 (1786113998) [ 217.077697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.091721] LNet: 11400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 217.103729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.139983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 220.480177] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:46:41 (1786114001) [ 220.576727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.598218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.640200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 220.901405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 223.989223] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:46:45 (1786114005) [ 224.021288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.033501] LNet: 12565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 224.036751] LNet: 12565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 224.048777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.096511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 224.287296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 224.486536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 227.857943] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:46:49 (1786114009) [ 227.902375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.924472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.971928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 231.483337] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:46:53 (1786114013) [ 231.557441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.589203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.633051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 231.652387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 231.856600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 231.879276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 232.095030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 232.115063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 232.281140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 232.307429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 232.535756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 232.562356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 232.728579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 232.745218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 232.910620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 232.932350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 233.110368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 233.126475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 233.286976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 233.306892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 233.472378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 233.487909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 233.644951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 233.663625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 233.834321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 233.863426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 234.067191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 234.091632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 237.447088] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:46:58 (1786114018) [ 237.547067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.564163] LNet: 14432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 237.572031] LNet: 14432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 237.585879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.632172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 237.659101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 241.100400] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:47:02 (1786114022) [ 241.156341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.184057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.249149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 241.280761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 244.989522] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:47:06 (1786114026) [ 245.025987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.057635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.128788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 245.156704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 249.142563] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:47:10 (1786114030) [ 249.201664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.237646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.299928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 249.323622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 253.396975] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:47:14 (1786114034) [ 253.501539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.556810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.608479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 253.640806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 257.614257] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:47:18 (1786114038) [ 257.674138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.697690] LNet: 17354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 257.701854] LNet: 17354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 257.715762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.774331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 257.796729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 261.500893] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:47:22 (1786114042) [ 261.681712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.707500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.758458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 262.002972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 266.196171] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:47:27 (1786114047) [ 266.274218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.310765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.464922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 266.664989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 270.598571] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:47:32 (1786114052) [ 270.714120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.734795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.796344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 270.938859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 274.833583] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:47:36 (1786114056) [ 274.917636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.950547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.224708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 275.380376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 279.406866] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:47:40 (1786114060) [ 279.468837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.501969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.559541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 279.598078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 283.428973] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:47:44 (1786114064) [ 283.530277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.563861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.621819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 283.652930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 287.390907] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:47:48 (1786114068) [ 287.431853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.465212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.506704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 290.768724] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:47:52 (1786114072) [ 290.893947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.916431] LNet: 22210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.918803] LNet: 22210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 290.928776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.966162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 294.523588] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:47:56 (1786114076) [ 294.570885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.593714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.657506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 294.854276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 298.422564] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:47:59 (1786114079) [ 298.524552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.568538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.622362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 298.649167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 302.499323] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:48:03 (1786114083) [ 302.550354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.570344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.621358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 302.646651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 306.169140] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:48:07 (1786114087) [ 306.218792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.251761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.310834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 306.332796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 309.647971] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:48:11 (1786114091) [ 309.711570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.738319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.767320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 309.784246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 313.450920] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:48:14 (1786114094) [ 313.628694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.659536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.696636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 313.749355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 313.766525] LNetError: 25739:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 313.791634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 313.825120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 317.426642] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:48:18 (1786114098) [ 317.523132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.550790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.588956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QbKK7P2CDL [ 317.631387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QbKK7P2CDL [ 317.662859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QbKK7P2CDL [ 320.777911] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:48:22 (1786114102) [ 320.872279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.889384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.914043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 320.931615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 320.951252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 323.717313] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:48:25 (1786114105) [ 323.811211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.848231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.881521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 323.903111] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 324.665214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 325.382907] Lustre: Lustre: Build Version: 2.17.55_1_gd52575f [ 327.103161] Key type lgssc registered [ 327.707324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 329.784464] LNetError: 27499:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.202.43@tcp99 has no available nets [ 331.177179] Key type lgssc unregistered [ 331.400247] LNetError: 28604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.420974] LNet: Removed LNI 192.168.202.43@tcp [ 331.741575] Key type .llcrypt unregistered [ 331.750371] Key type ._llcrypt unregistered [ 334.966686] Key type ._llcrypt registered [ 334.968411] Key type .llcrypt registered [ 335.723606] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:48:37 (1786114117) [ 335.954248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.960894] alg: No test for adler32 (adler32-zlib) [ 336.807300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.839873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 338.147051] LNet: Message forwarding will be enabled in 2s [ 341.215138] LNet: Message forwarding enabled [ 341.433214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 341.443223] LNet: Message forwarding will stop in 2s to 6s [ 345.775786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 346.270876] LNet: Message forwarding disabled [ 347.046109] LNet: Message forwarding will be enabled in 2s [ 350.369377] LNet: Message forwarding enabled [ 351.304908] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:48:52 (1786114132) [ 351.402191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.414850] LNet: 30008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.197248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.247825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 356.418482] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:48:57 (1786114137) [ 356.553689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.581971] LNet: 30593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.599166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.641201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 356.660568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 356.679859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 356.703673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 356.723566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 356.741237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 356.756211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 356.779627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 356.799408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 356.827253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 356.856694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 356.883871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 356.916808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 356.941730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 356.969122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 356.999698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 357.027424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 357.053299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 357.097843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 357.117884] LNetError: 30637:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 360.825914] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:49:02 (1786114142) [ 361.154966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.171259] LNet: 31208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.186471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.241965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 365.459567] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:49:06 (1786114146) [ 366.419876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg243-server [ 367.424267] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:49:08 (1786114148) [ 367.516374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.528616] LNet: 32139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.539804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.568176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 367.580072] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 368.035224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[44-48]@tcp [ 368.048524] LNet: 32272:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 368.082077] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 368.094027] LNetError: 32143:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.44@tcp has gone from up to down [ 369.126378] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 369.133271] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 370.156800] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 370.167118] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 371.167409] LNetError: 32147:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.46@tcp has gone from up to down [ 371.179971] LNetError: 32147:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 371.501563] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:49:12 (1786114152) [ 372.209635] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 372.219077] LNetError: 32143:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 372.425372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg243-server [ 373.336619] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:49:14 (1786114154) [ 373.483773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.239512] LNetError: 32142:0:(router.c:1265:lnet_check_routers()) Failed to discover router 192.168.202.44@tcp [ 374.559776] LNet: 33207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.563929] LNetError: 33207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.587128] LNet: Removed LNI 192.168.202.43@tcp [ 374.607603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.653772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 374.680658] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 375.127703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[44-49/2]@tcp [ 375.146375] LNet: 33325:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 375.152545] LNet: 33325:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 375.170215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[44-49/2]@tcp [ 375.182641] LNetError: 33212:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.48@tcp has gone from up to down [ 378.836752] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:49:20 (1786114160) [ 378.882019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.895360] LNetError: 33911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.910701] LNet: Removed LNI 192.168.202.43@tcp [ 378.935230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.968398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 378.996707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.015150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.043312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.076490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.100433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.136377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 379.169835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.183987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.226540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.284432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.311223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.347458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 379.401479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.433442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.473379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.522243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.544363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.581252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 379.620504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.640645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.674832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.712521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.731358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.761202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 379.808127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.829919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.870858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.902781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.921077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.947600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 379.974715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.996360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.032145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 383.464875] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:49:24 (1786114164) [ 384.356835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg243-server [ 385.212710] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:49:26 (1786114166) [ 386.248350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg243-server [ 387.165559] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:49:28 (1786114168) [ 387.224600] LNet: 35308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.228795] LNet: 35308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 387.614531] Key type .llcrypt unregistered [ 387.616551] Key type ._llcrypt unregistered [ 391.070139] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 391.129940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 391.132196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 391.134659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 393.457182] Key type ._llcrypt registered [ 393.459214] Key type .llcrypt registered [ 393.538211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 393.546446] alg: No test for adler32 (adler32-zlib) [ 394.368835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.396840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 394.409540] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 394.722703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 394.732751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 395.848386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 395.861670] LNet: Removed LNI 10.1.2.3@tcp [ 395.920608] LNet: 36038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.927409] LNetError: 36038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.946939] LNet: Removed LNI 192.168.202.43@tcp [ 396.299509] Key type .llcrypt unregistered [ 396.301025] Key type ._llcrypt unregistered [ 402.276945] Key type ._llcrypt registered [ 402.278848] Key type .llcrypt registered [ 402.961163] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:49:44 (1786114184) [ 403.224575] Key type .llcrypt unregistered [ 403.227332] Key type ._llcrypt unregistered [ 405.799796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 405.889969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 405.893714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 405.897535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 408.183781] Key type ._llcrypt registered [ 408.185411] Key type .llcrypt registered [ 408.303696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 408.317438] alg: No test for adler32 (adler32-zlib) [ 409.159188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.207220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 409.223300] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 409.655085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 409.670444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 411.002766] LNet: 37524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.008867] LNetError: 37524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 411.011616] LNet: Removed LNI 10.1.2.3@tcp [ 411.017318] LNet: Removed LNI 192.168.202.43@tcp [ 411.337561] Key type .llcrypt unregistered [ 411.340183] Key type ._llcrypt unregistered [ 416.270522] Key type ._llcrypt registered [ 416.271667] Key type .llcrypt registered [ 416.912881] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:49:58 (1786114198) [ 417.125561] Key type .llcrypt unregistered [ 417.128344] Key type ._llcrypt unregistered [ 419.725234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 419.768787] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 419.771660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 419.775163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 422.127924] Key type ._llcrypt registered [ 422.131194] Key type .llcrypt registered [ 422.241164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 422.250980] alg: No test for adler32 (adler32-zlib) [ 423.088467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.119493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 423.130115] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 424.297792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 424.314407] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 425.468707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 425.495243] LNet: Removed LNI 10.1.2.3@tcp [ 425.511083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 425.547888] LNet: Removed LNI 10.1.2.31@tcp [ 425.559926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 425.581583] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 426.000551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 426.013325] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 426.572444] LNet: 39437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.575708] LNetError: 39437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 426.579528] LNet: Removed LNI 10.1.2.31@tcp [ 426.589592] LNet: Removed LNI 10.1.2.3@tcp [ 426.966628] Key type .llcrypt unregistered [ 426.971072] Key type ._llcrypt unregistered [ 433.316538] Key type ._llcrypt registered [ 433.318167] Key type .llcrypt registered [ 434.136733] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:50:15 (1786114215) [ 434.479200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 434.488161] alg: No test for adler32 (adler32-zlib) [ 435.369249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.421366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 435.440352] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 435.922792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.932119] LNet: 40389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.934827] LNetError: 40389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.942869] LNet: Removed LNI 192.168.202.43@tcp [ 435.954625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.981696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.43@tcp [ 435.994803] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 436.472293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.43@tcp [ 436.491937] LNet: Removed LNI 192.168.202.43@tcp [ 436.530663] LNet: 40524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.905342] Key type .llcrypt unregistered [ 436.906621] Key type ._llcrypt unregistered [ 443.444925] Key type ._llcrypt registered [ 443.446509] Key type .llcrypt registered [ 444.212503] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:50:25 (1786114225) [ 445.442766] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg243-server [ 446.262364] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:50:27 (1786114227) [ 446.521407] Key type .llcrypt unregistered [ 446.523347] Key type ._llcrypt unregistered [ 449.875707] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 450.001420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 450.011092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 450.018620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 452.442039] Key type ._llcrypt registered [ 452.443676] Key type .llcrypt registered [ 452.593468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 452.605933] alg: No test for adler32 (adler32-zlib) [ 453.479856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.517675] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 453.965985] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 455.304910] LNet: 42359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.309572] LNetError: 42359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 455.312672] LNet: Removed LNI 192.168.202.43@tcp1 [ 455.322469] LNet: Removed LNI 10.1.2.3@tcp2 [ 455.702287] Key type .llcrypt unregistered [ 455.703320] Key type ._llcrypt unregistered [ 462.384880] Key type ._llcrypt registered [ 462.386610] Key type .llcrypt registered [ 463.137657] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:50:44 (1786114244) [ 463.409711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 463.428329] alg: No test for adler32 (adler32-zlib) [ 464.317842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.349483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 464.365976] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 464.942119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.951897] LNet: 43328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.954583] LNetError: 43328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.961584] LNet: Removed LNI 192.168.202.43@tcp [ 464.972389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.008938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 465.025389] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 468.879527] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:50:50 (1786114250) [ 468.923303] LNet: 43912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.930453] LNetError: 43912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.941791] LNet: Removed LNI 192.168.202.43@tcp [ 469.288680] Key type .llcrypt unregistered [ 469.290609] Key type ._llcrypt unregistered [ 472.489905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 472.638055] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 472.642947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 472.647465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 474.988259] Key type ._llcrypt registered [ 474.994836] Key type .llcrypt registered [ 475.094566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 475.113642] alg: No test for adler32 (adler32-zlib) [ 475.958866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.989145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 476.011490] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 476.446348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 476.454504] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 477.701798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.717843] LNet: 44655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.721246] LNetError: 44655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 477.723325] LNet: Removed LNI 10.1.2.3@tcp [ 477.732943] LNet: Removed LNI 192.168.202.43@tcp [ 477.749766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.786287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 477.804634] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 477.808419] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 478.062305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.075691] LNet: Removed LNI 10.1.2.3@tcp [ 478.086773] LNet: Removed LNI 192.168.202.43@tcp [ 478.101467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.144470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 478.165649] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 478.168967] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 478.415859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 478.426204] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 478.950729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.962469] LNet: 44844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.964894] LNet: 44844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 478.968201] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 478.971283] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 478.975767] LNet: Removed LNI 10.1.2.3@tcp [ 478.977325] LNet: Removed LNI 192.168.202.43@tcp [ 479.003658] LNet: Removed LNI 192.168.202.43@tcp2 [ 479.024573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.076719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 479.095307] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 479.099155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 479.103458] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 482.761590] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:51:04 (1786114264) [ 482.849924] LNet: 45437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.861578] LNetError: 45437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 482.871124] LNetError: 45437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 482.882744] LNet: Removed LNI 10.1.2.3@tcp [ 482.887515] LNet: Removed LNI 192.168.202.43@tcp [ 482.899064] LNet: Removed LNI 192.168.202.43@tcp2 [ 483.261729] Key type .llcrypt unregistered [ 483.263047] Key type ._llcrypt unregistered [ 486.780503] Key type ._llcrypt registered [ 486.784145] Key type .llcrypt registered [ 486.929858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.938639] alg: No test for adler32 (adler32-zlib) [ 487.805951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.839286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 487.854841] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 488.378391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.399140] LNet: 45835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 488.407769] LNetError: 45835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 488.433803] LNet: Removed LNI 192.168.202.43@tcp [ 488.460658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.508351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 488.530609] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 488.811147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.830453] LNet: Removed LNI 192.168.202.43@tcp [ 488.840965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.898467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 488.921321] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 489.176578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.190167] LNet: 45895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.195159] LNet: 45895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 489.199465] LNetError: 45895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.203874] LNetError: 45895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 489.212460] LNet: Removed LNI 192.168.202.43@tcp [ 489.224816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.277740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 489.300452] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 489.526230] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 489.623379] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 489.629746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 489.634928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 491.822291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.836726] LNet: 45975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.841692] LNetError: 45975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.855710] LNet: Removed LNI 192.168.202.43@tcp [ 491.876688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.909099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 491.932295] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 495.373179] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:51:16 (1786114276) [ 495.421345] LNet: 46562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.424601] LNetError: 46562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 495.436922] LNet: Removed LNI 192.168.202.43@tcp [ 495.794642] Key type .llcrypt unregistered [ 495.799144] Key type ._llcrypt unregistered [ 498.670465] Key type ._llcrypt registered [ 498.672075] Key type .llcrypt registered [ 498.770125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.778036] alg: No test for adler32 (adler32-zlib) [ 499.614766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.640585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 499.651768] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 500.065086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.078543] LNet: 46969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.080904] LNetError: 46969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.094868] LNet: Removed LNI 192.168.202.43@tcp [ 500.104272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.130964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 500.144092] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 503.499472] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:51:25 (1786114285) [ 503.546304] LNet: 47553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.558640] LNetError: 47553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.569227] LNet: Removed LNI 192.168.202.43@tcp [ 503.939445] Key type .llcrypt unregistered [ 503.940960] Key type ._llcrypt unregistered [ 506.928358] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 506.987452] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 506.993554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 507.000754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 509.327720] Key type ._llcrypt registered [ 509.329021] Key type .llcrypt registered [ 509.451638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 509.460586] alg: No test for adler32 (adler32-zlib) [ 510.278257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.307904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 510.328934] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 510.817107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.830760] LNet: 48006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.833626] LNetError: 48006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 510.845692] LNet: Removed LNI 192.168.202.43@tcp [ 510.862729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.894761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 510.911891] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 514.280648] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:51:35 (1786114295) [ 514.335539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.351074] LNet: 48600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.355716] LNetError: 48600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.365562] LNet: Removed LNI 192.168.202.43@tcp [ 514.385603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.440678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 517.380715] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:51:38 (1786114298) [ 517.468230] LNet: 49185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.772503] Key type .llcrypt unregistered [ 517.773408] Key type ._llcrypt unregistered [ 520.455736] Key type ._llcrypt registered [ 520.457502] Key type .llcrypt registered [ 520.540732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 520.549286] alg: No test for adler32 (adler32-zlib) [ 521.364797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.394137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.405703] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 521.776612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.785983] LNet: 49582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.791589] LNetError: 49582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.800516] LNet: Removed LNI 192.168.202.43@tcp [ 521.808346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.835782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 521.846765] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 524.852328] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:51:46 (1786114306) [ 524.948617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.966715] LNet: 50173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 524.973649] LNetError: 50173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 524.995881] LNet: Removed LNI 192.168.202.43@tcp [ 525.011946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.047785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 528.320621] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:51:49 (1786114309) [ 529.138805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 529.962721] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:51:51 (1786114311) [ 530.665846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 531.486346] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:51:53 (1786114313) [ 532.204311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 532.942797] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:51:54 (1786114314) [ 533.634981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 534.256126] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:51:55 (1786114315) [ 534.904623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 535.552803] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:51:57 (1786114317) [ 535.594668] LNet: 52217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 535.870555] Key type .llcrypt unregistered [ 535.871559] Key type ._llcrypt unregistered [ 538.202401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 538.253958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 538.261774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 538.265221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 540.531099] Key type ._llcrypt registered [ 540.532695] Key type .llcrypt registered [ 540.612455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 540.623298] alg: No test for adler32 (adler32-zlib) [ 541.434175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.477551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 541.489615] LNet: Added LNI 192.168.202.43@tcp [8/512/0/180] [ 541.495620] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 541.496911] LNet: 52545:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 541.545943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.557456] LNet: 52564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.561621] LNetError: 52564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 541.565544] LNet: Removed LNI 10.1.2.3@tcp [ 541.567063] LNetError: 52564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.575691] LNet: Removed LNI 192.168.202.43@tcp [ 541.584831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.613016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 541.623920] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 541.657734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.667868] LNet: 52588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.671186] LNetError: 52588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.683256] LNet: Removed LNI 192.168.202.43@tcp [ 541.696585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.738455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 541.756820] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 541.760498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 541.763767] LNet: 52596:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 541.807484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.818340] LNet: 52614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.821357] LNetError: 52614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 541.824315] LNet: Removed LNI 10.1.2.3@tcp [ 541.825477] LNetError: 52614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.834321] LNet: Removed LNI 192.168.202.43@tcp [ 541.847286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.878771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 541.887952] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 541.894939] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 541.897041] LNet: 52622:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 541.973993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.990532] LNet: 52648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.995386] LNetError: 52648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 541.999629] LNet: Removed LNI 10.1.2.3@tcp [ 542.001453] LNetError: 52648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.012081] LNet: Removed LNI 192.168.202.43@tcp [ 542.025896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.056894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 542.071227] LNet: Added LNI 192.168.202.43@tcp [8/256/0/182] [ 542.073834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 542.112394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.124558] LNet: 52676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.127283] LNetError: 52676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 542.130180] LNet: Removed LNI 10.1.2.3@tcp [ 542.131378] LNetError: 52676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.140403] LNet: Removed LNI 192.168.202.43@tcp [ 542.150624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.183136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 542.194213] LNet: Added LNI 192.168.202.43@tcp [8/256/0/182] [ 542.196562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 542.197934] LNet: 52684:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 542.230126] LNet: Removed LNI 10.1.2.3@tcp [ 542.236672] LNet: Removed LNI 192.168.202.43@tcp [ 542.550755] Key type .llcrypt unregistered [ 542.552232] Key type ._llcrypt unregistered [ 547.261132] Key type ._llcrypt registered [ 547.262511] Key type .llcrypt registered [ 547.840907] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:52:09 (1786114329) [ 548.036222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 548.043695] alg: No test for adler32 (adler32-zlib) [ 548.866332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.903628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 548.918806] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 549.273596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.43 [ 549.283284] LNet: 53659:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 549.288449] LNet: Removed LNI 192.168.202.43@tcp [ 551.893932] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:52:13 (1786114333) [ 551.941782] LNet: 54220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.946320] LNetError: 54220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.957441] LNet: Removed LNI 192.168.202.43@tcp [ 552.288530] Key type .llcrypt unregistered [ 552.290071] Key type ._llcrypt unregistered [ 554.723371] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 554.837542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 557.162653] Key type ._llcrypt registered [ 557.164684] Key type .llcrypt registered [ 557.263170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 557.270809] alg: No test for adler32 (adler32-zlib) [ 558.249517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 561.206817] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:52:22 (1786114342) [ 561.277373] LNet: 55138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.285583] LNetError: 55138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 561.295940] LNet: Removed LNI 10.1.2.3@tcp [ 561.579615] Key type .llcrypt unregistered [ 561.582268] Key type ._llcrypt unregistered [ 564.320921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 564.430985] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 564.433768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 564.437066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 566.824308] Key type ._llcrypt registered [ 566.825729] Key type .llcrypt registered [ 566.935619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 566.947343] alg: No test for adler32 (adler32-zlib) [ 567.784139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 570.594677] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:52:32 (1786114352) [ 571.273783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 571.982305] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:52:33 (1786114353) [ 572.048000] LNet: 56322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.052854] LNetError: 56322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 572.060370] LNet: Removed LNI 10.1.2.3@tcp [ 572.330392] Key type .llcrypt unregistered [ 572.331485] Key type ._llcrypt unregistered [ 574.505540] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 574.633666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 574.640782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 574.644958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 576.957692] Key type ._llcrypt registered [ 576.959107] Key type .llcrypt registered [ 577.043836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.051127] alg: No test for adler32 (adler32-zlib) [ 577.863321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.894352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 581.580680] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:52:43 (1786114363) [ 581.736259] LNet: 57550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.742541] LNetError: 57550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 581.756159] LNet: Removed LNI 10.1.2.3@tcp [ 582.004110] Key type .llcrypt unregistered [ 582.005613] Key type ._llcrypt unregistered [ 584.311375] Key type ._llcrypt registered [ 584.313473] Key type .llcrypt registered [ 584.411742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.421258] alg: No test for adler32 (adler32-zlib) [ 585.245219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 585.259916] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 588.861817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 589.173519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_interrupt -r 1 [ 589.191989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_interrupt -r 1 [ 589.213731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 589.225057] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 589.323936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 589.339335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 589.343065] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 589.353667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 589.366571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.382134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.396745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 589.417565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 589.434119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 589.441335] LNet: 57987:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 589.452015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 589.475411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 589.489854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 589.514612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 589.541685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 589.572613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 589.608102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 589.880465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_dropped -r 1 [ 589.898671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_dropped -r 1 [ 589.914575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 589.927768] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 590.037843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 590.051452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 590.070263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 590.083711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 590.095763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 590.108763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.126937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.149391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.169399] LNet: 58096:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 590.187275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.213662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.237537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.252586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.273016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.297507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 590.320154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 590.609734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_aborted -r 1 [ 590.625197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_aborted -r 1 [ 590.641158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 590.742480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 590.756753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 590.773354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 590.795627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 590.811957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 590.838772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 590.860428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 590.883818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 590.902339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 590.919705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 590.939046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 590.956034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 590.971365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 590.988623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 591.010384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 591.242686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_no_route -r 1 [ 591.262289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_no_route -r 1 [ 591.282465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 591.291200] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 591.295940] LNet: Skipped 1 previous similar message [ 591.411554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 591.430695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 591.445815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 591.464731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 591.481564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 591.498430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 591.515326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 591.532989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 591.543558] LNet: 58314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 591.546619] LNet: 58314:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 591.555445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 591.570191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 591.585744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 591.600571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 591.617314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 591.631094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 591.643987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 591.922612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_timeout -r 1 [ 591.941417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_timeout -r 1 [ 591.960839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 592.073603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 592.091558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 592.111371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 592.127883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 592.145511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 592.168316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 592.192039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 592.213363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 592.230457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.248909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.264922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.280673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.299703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.315223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.335876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 592.562281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_error -r 1 [ 592.585642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_error -r 1 [ 592.602945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 592.722092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 592.737557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 592.752993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 592.767600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 592.780275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 592.793243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 592.811312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 592.825929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 592.840754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 592.858996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 592.881587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 592.898586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 592.914943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 592.932079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 592.949606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.054737] LNet: 58573:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.057672] LNetError: 58573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.066081] LNet: Removed LNI 192.168.202.43@tcp [ 593.229845] Key type .llcrypt unregistered [ 593.231417] Key type ._llcrypt unregistered [ 600.361591] Key type ._llcrypt registered [ 600.363290] Key type .llcrypt registered [ 600.930315] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:53:02 (1786114382) [ 601.341375] Key type .llcrypt unregistered [ 601.343887] Key type ._llcrypt unregistered [ 603.789861] Key type ._llcrypt registered [ 603.791502] Key type .llcrypt registered [ 603.890228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 603.901222] alg: No test for adler32 (adler32-zlib) [ 604.725560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 604.741300] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 608.665957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 612.871605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.885623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 612.894722] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 613.496878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_interrupt -r 1 [ 613.517477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_interrupt -r 1 [ 613.538944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_interrupt -r 1 [ 613.554892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_interrupt -r 1 [ 613.571059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_interrupt -r 1 [ 613.592767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_interrupt -r 1 [ 613.612346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_interrupt -r 1 [ 613.628859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_interrupt -r 1 [ 613.648545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 613.658776] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 613.662333] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 613.756458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 613.770318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 613.785873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 613.801661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 613.817547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 613.833138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 613.848818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 613.868739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 613.879784] LNet: 60036:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 613.889550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 613.905750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 613.921127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 613.937845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 613.952747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 613.965842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 613.979303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 614.210766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_dropped -r 1 [ 614.231432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_dropped -r 1 [ 614.259975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_dropped -r 1 [ 614.279988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_dropped -r 1 [ 614.297786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_dropped -r 1 [ 614.314470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_dropped -r 1 [ 614.331348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_dropped -r 1 [ 614.349889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_dropped -r 1 [ 614.369129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 614.379522] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 614.382875] LNet: Skipped 2 previous similar messages [ 614.483492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 614.498446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 614.515709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 614.529643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 614.543624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 614.557651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 614.573275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 614.587303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 614.595734] LNet: 60158:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 614.604814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 614.628587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 614.646793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 614.662856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 614.680950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 614.700268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 614.719326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 614.998112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_aborted -r 1 [ 615.022277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_aborted -r 1 [ 615.042601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_aborted -r 1 [ 615.062153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_aborted -r 1 [ 615.080943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_aborted -r 1 [ 615.106033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_aborted -r 1 [ 615.128438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_aborted -r 1 [ 615.146476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_aborted -r 1 [ 615.166391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 615.278697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 615.297801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 615.319720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 615.340868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 615.363851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 615.383621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 615.418623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 615.445973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 615.478196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 615.499861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 615.525354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 615.546209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 615.565954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 615.585953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 615.607482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 615.912800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_no_route -r 1 [ 615.950297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_no_route -r 1 [ 615.978483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_no_route -r 1 [ 616.009470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_no_route -r 1 [ 616.035766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_no_route -r 1 [ 616.060600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_no_route -r 1 [ 616.083630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_no_route -r 1 [ 616.107601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_no_route -r 1 [ 616.125625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 616.138147] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 616.142189] LNet: Skipped 7 previous similar messages [ 616.302141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 616.327422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 616.358714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 616.382785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 616.406917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 616.441097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 616.471455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 616.492570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 616.506542] LNet: 60401:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 616.511859] LNet: 60401:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 616.521911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 616.539187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.557451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 616.577855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 616.593444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 616.617921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 616.638859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 616.887800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_timeout -r 1 [ 616.913891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_timeout -r 1 [ 616.938590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_timeout -r 1 [ 616.957426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_timeout -r 1 [ 616.974984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_timeout -r 1 [ 616.995425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_timeout -r 1 [ 617.014757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_timeout -r 1 [ 617.034793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_timeout -r 1 [ 617.057702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 617.182522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 617.198730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 617.220753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 617.236741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 617.252421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 617.269739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 617.290753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 617.309353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 617.329389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 617.354882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 617.376252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 617.398928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 617.420354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 617.442416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 617.471053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 617.746643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_error -r 1 [ 617.766194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_error -r 1 [ 617.787861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_error -r 1 [ 617.804562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_error -r 1 [ 617.821713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_error -r 1 [ 617.841561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_error -r 1 [ 617.859474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_error -r 1 [ 617.881037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_error -r 1 [ 617.899858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 618.008918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 618.024901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 618.040805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 618.054462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 618.074845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 618.089656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 618.110692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 618.130393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 618.151103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 618.175570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 618.189027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 618.204086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 618.217707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 618.234439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 618.253354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 620.956861] LNet: 60719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.960158] LNetError: 60719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.964724] LNet: Removed LNI 192.168.202.43@tcp [ 621.992526] LNet: Removed LNI 192.168.202.43@tcp1 [ 622.190505] Key type .llcrypt unregistered [ 622.191519] Key type ._llcrypt unregistered [ 630.215821] Key type ._llcrypt registered [ 630.217289] Key type .llcrypt registered [ 630.873761] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:53:32 (1786114412) [ 631.278360] Key type .llcrypt unregistered [ 631.279540] Key type ._llcrypt unregistered [ 634.035239] Key type ._llcrypt registered [ 634.036216] Key type .llcrypt registered [ 634.107899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.115371] alg: No test for adler32 (adler32-zlib) [ 634.949228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 634.963911] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 638.827491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 639.126073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_dropped -r 1 [ 639.147634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_dropped -r 1 [ 639.168462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 639.178345] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 639.279964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 639.295627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 639.313688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 639.329853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 639.353116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 639.368861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 639.386133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 639.399463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 639.408547] LNet: 61965:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 639.424293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 639.443026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.457648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 639.476546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 639.495254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 639.511821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 639.531155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 639.778252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_error -r 1 [ 639.795889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_error -r 1 [ 639.813849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 639.824895] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 639.918375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 639.929349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 639.942114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 639.956407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 639.974269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 639.996518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 640.014086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 640.030269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 640.038550] LNet: 62075:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 640.048593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 640.064957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 640.081113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 640.100936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 640.124443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.141341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.159700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 640.372690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_timeout -r 1 [ 640.392748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_timeout -r 1 [ 640.409759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 640.534329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 640.550430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 640.568490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 640.582693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 640.597591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 640.611877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 640.625577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 640.639653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 640.655712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 640.673457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 640.690232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 640.704432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 640.718456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 640.732816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 640.747926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 640.860138] LNet: 62225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.863959] LNetError: 62225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.875372] LNet: Removed LNI 192.168.202.43@tcp [ 641.074521] Key type .llcrypt unregistered [ 641.076765] Key type ._llcrypt unregistered [ 647.910239] Key type ._llcrypt registered [ 647.912155] Key type .llcrypt registered [ 648.579245] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:53:50 (1786114430) [ 648.950274] Key type .llcrypt unregistered [ 648.951674] Key type ._llcrypt unregistered [ 651.655318] Key type ._llcrypt registered [ 651.656425] Key type .llcrypt registered [ 651.799654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 651.815945] alg: No test for adler32 (adler32-zlib) [ 652.663073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 652.679975] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 656.556349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 660.629993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.655836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 660.673156] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 661.420434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_dropped -r 1 [ 661.440807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e remote_dropped -r 1 [ 661.461172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_dropped -r 1 [ 661.477854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e remote_dropped -r 1 [ 661.493273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e remote_dropped -r 1 [ 661.511569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e remote_dropped -r 1 [ 661.528553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e remote_dropped -r 1 [ 661.547639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e remote_dropped -r 1 [ 661.571188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 661.583363] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 661.592512] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.143@tcp [ 661.696747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 661.715306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 661.734844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 661.750069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 661.767605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 661.779927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 661.798305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 661.816691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 661.835702] LNet: 63688:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 661.847936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.869885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.890228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.907517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.929862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.950181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.967868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 661.986982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 662.241484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_error -r 1 [ 662.261957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e remote_error -r 1 [ 662.281628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_error -r 1 [ 662.297119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e remote_error -r 1 [ 662.313448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e remote_error -r 1 [ 662.329895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e remote_error -r 1 [ 662.347245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e remote_error -r 1 [ 662.364970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e remote_error -r 1 [ 662.389784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 662.401453] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 662.404451] LNet: Skipped 2 previous similar messages [ 662.509265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 662.535302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 662.550446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 662.564531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 662.583724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 662.599759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 662.618966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 662.642685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 662.652371] LNet: 63811:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 662.662887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 662.682181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 662.699614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 662.714515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 662.736644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 662.762692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 662.788682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 662.818355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 663.081164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_timeout -r 1 [ 663.105752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e remote_timeout -r 1 [ 663.128437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_timeout -r 1 [ 663.152611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e remote_timeout -r 1 [ 663.172738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e remote_timeout -r 1 [ 663.194533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e remote_timeout -r 1 [ 663.215550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e remote_timeout -r 1 [ 663.236128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e remote_timeout -r 1 [ 663.258745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.143@tcp [ 663.419329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 663.438884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 663.455600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 663.470381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 663.482623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 663.498417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 663.513112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 663.528413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 663.542861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 663.556815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 663.577295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 663.593818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 663.612842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 663.629841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 663.649836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 663.666732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 666.256707] LNet: 64012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.261216] LNetError: 64012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.295240] LNet: Removed LNI 192.168.202.43@tcp [ 667.298565] LNetError: 64012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.313412] LNet: Removed LNI 192.168.202.43@tcp1 [ 667.520742] Key type .llcrypt unregistered [ 667.524637] Key type ._llcrypt unregistered [ 675.631236] Key type ._llcrypt registered [ 675.632682] Key type .llcrypt registered [ 676.262646] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:54:17 (1786114457) [ 677.016442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 677.854576] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:54:19 (1786114459) [ 678.240524] Key type .llcrypt unregistered [ 678.246682] Key type ._llcrypt unregistered [ 681.133150] Key type ._llcrypt registered [ 681.134312] Key type .llcrypt registered [ 681.238227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.257585] alg: No test for adler32 (adler32-zlib) [ 682.078571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 682.090763] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 685.891806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 686.208618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e network_timeout -r 1 [ 686.237188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e network_timeout -r 1 [ 686.260598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 686.272613] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 686.371290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 686.385529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 686.399653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 686.412596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 686.428791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 686.442669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 686.457269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 686.472479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 686.481963] LNet: 65553:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 686.493343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 686.516094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 686.534771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 686.550441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 686.570335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 686.590845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 686.742605] LNet: 65592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.748464] LNetError: 65592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.757604] LNet: Removed LNI 192.168.202.43@tcp [ 686.939911] Key type .llcrypt unregistered [ 686.944580] Key type ._llcrypt unregistered [ 694.792472] Key type ._llcrypt registered [ 694.796650] Key type .llcrypt registered [ 694.915861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 694.924781] alg: No test for adler32 (adler32-zlib) [ 695.767096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 695.787060] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 699.863712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 703.496620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.518547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 703.533345] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 704.210556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e network_timeout -r 1 [ 704.228402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e network_timeout -r 1 [ 704.250684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e network_timeout -r 1 [ 704.270918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e network_timeout -r 1 [ 704.288376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e network_timeout -r 1 [ 704.308487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e network_timeout -r 1 [ 704.325875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e network_timeout -r 1 [ 704.345662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e network_timeout -r 1 [ 704.363756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 704.373941] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 704.483359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 704.497448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 704.512239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 704.528905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 704.545496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 704.563257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 704.578739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 704.605704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 704.614014] LNet: 66468:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 704.623895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 704.640940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 704.661270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 704.683892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 704.703610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 704.717962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 707.387613] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 707.533053] LNet: 66542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.535705] LNetError: 66542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.575193] LNet: Removed LNI 192.168.202.43@tcp [ 708.577970] LNetError: 66542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.592205] LNet: Removed LNI 192.168.202.43@tcp1 [ 708.838082] Key type .llcrypt unregistered [ 708.839749] Key type ._llcrypt unregistered [ 716.832479] Key type ._llcrypt registered [ 716.833941] Key type .llcrypt registered [ 717.539557] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:54:58 (1786114498) [ 717.713423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.719214] alg: No test for adler32 (adler32-zlib) [ 718.540995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.563236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.575580] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 718.952734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 718.964759] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 719.405887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 719.445304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 719.476586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 719.493676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 719.515354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 719.538533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 719.558387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.43@tcp [ 719.569453] LNetError: 67640:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.43@tcp to recovery [ 719.584653] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 719.585515] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 721.632320] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 721.778874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 725.727992] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 726.086165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 726.106839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 726.144864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.154696] LNet: 67747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.158950] LNetError: 67747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.161910] LNet: Removed LNI 192.168.202.43@tcp [ 726.167403] LNet: Removed LNI 192.168.202.43@tcp1 [ 726.178858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.213614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.225346] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 726.566917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 726.575320] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 726.952451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 726.977053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 727.001426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 727.022708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 727.043469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 727.062795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 727.078571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 727.094141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.43@tcp [ 727.103133] LNetError: 68020:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.43@tcp to recovery [ 729.247374] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 729.253048] LNet: Skipped 1 previous similar message [ 730.329617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 733.343323] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 737.720981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 737.736590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 737.751838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 740.985842] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:55:22 (1786114522) [ 741.047883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.066296] LNet: 68734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.070077] LNetError: 68734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.073911] LNetError: 68734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 741.077755] LNet: Removed LNI 192.168.202.43@tcp [ 741.089901] LNet: Removed LNI 192.168.202.43@tcp1 [ 741.113573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.142281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.160230] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 741.566339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 741.576438] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 742.013386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 742.046246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 742.064830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 742.097412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 742.115869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 742.132606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 742.149441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.43@tcp --health 0 [ 742.176080] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.43@tcp [ 742.197165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 743.199832] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 743.202976] LNet: Skipped 1 previous similar message [ 757.823506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 757.887325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 757.902418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 757.917468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.43@tcp --health 500 [ 757.965459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 764.244547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 764.289398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.300577] LNet: 69238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.304770] LNetError: 69238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.308577] LNetError: 69238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 764.311822] LNet: Removed LNI 192.168.202.43@tcp [ 764.319464] LNet: Removed LNI 192.168.202.43@tcp1 [ 764.328483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.354348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.364499] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 764.693595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 764.705259] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 765.097975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 765.126497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 765.160409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 765.176954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 765.196196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 765.214826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 765.232114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 765.246770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.43@tcp --health 0 [ 766.369050] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 766.374858] LNet: Skipped 3 previous similar messages [ 767.434469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 776.976256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 776.991080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 777.004962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 779.828514] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:56:01 (1786114561) [ 780.008413] LNet: 70295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.012672] LNetError: 70295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.016112] LNetError: 70295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 780.020038] LNet: Removed LNI 192.168.202.43@tcp [ 780.026631] LNet: Removed LNI 192.168.202.43@tcp1 [ 780.301777] Key type .llcrypt unregistered [ 780.303802] Key type ._llcrypt unregistered [ 782.546390] Key type ._llcrypt registered [ 782.547443] Key type .llcrypt registered [ 782.628794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 782.636050] alg: No test for adler32 (adler32-zlib) [ 783.459780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 783.475810] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 787.762953] Lustre: DEBUG MARKER: Initial discovery [ 787.868419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.143@tcp [ 788.971243] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 789.037709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 789.055237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.143@tcp [ 789.065447] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 789.080567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 789.098190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.143@tcp -r 1 -m GET -l 3 [ 789.115943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.143@tcp -r 1 -m GET -e local_error [ 789.137717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.143@tcp -r 1 -m PUT -l 6 [ 789.599108] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 789.795912] Lustre: DEBUG MARKER: Force oleg243-client.virtnet to discover 192.168.202.143@tcp (in background) [ 789.843183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.143@tcp [ 790.447156] Lustre: DEBUG MARKER: Force oleg243-server to discover 192.168.202.43@tcp [ 791.470284] Lustre: DEBUG MARKER: Wait for 71050 [ 792.863210] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 793.460929] Lustre: DEBUG MARKER: Finished wait on 71050 [ 793.499881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 793.519127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 793.632666] LNet: 71504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.635288] LNetError: 71504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.647406] LNet: Removed LNI 192.168.202.43@tcp [ 793.828443] Key type .llcrypt unregistered [ 793.829593] Key type ._llcrypt unregistered [ 800.732955] Key type ._llcrypt registered [ 800.734273] Key type .llcrypt registered [ 801.295259] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:56:22 (1786114582) [ 801.496439] Key type .llcrypt unregistered [ 801.497551] Key type ._llcrypt unregistered [ 803.801780] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.908746] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.912628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.916563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 806.149285] Key type ._llcrypt registered [ 806.150958] Key type .llcrypt registered [ 806.226046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.232130] alg: No test for adler32 (adler32-zlib) [ 807.050968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.080639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.099116] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 807.444619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.453619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.422308] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:56:32 (1786114592) [ 811.482884] LNet: 73575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.487359] LNetError: 73575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.491325] LNet: Removed LNI 10.1.2.3@tcp [ 811.500934] LNet: Removed LNI 192.168.202.43@tcp [ 811.800444] Key type .llcrypt unregistered [ 811.802192] Key type ._llcrypt unregistered [ 814.858371] Key type ._llcrypt registered [ 814.859718] Key type .llcrypt registered [ 814.963219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.971689] alg: No test for adler32 (adler32-zlib) [ 815.799426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.840844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.859638] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 816.252116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 816.260996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.801735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.43@tcp [ 817.824491] LNetError: Unexpected error -22 connecting to 192.168.202.43@tcp at host 192.168.202.43:988 [ 819.599511] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:56:41 (1786114601) [ 819.675530] LNet: 74711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.681228] LNetError: 74711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 819.688305] LNet: Removed LNI 10.1.2.3@tcp [ 819.698694] LNet: Removed LNI 192.168.202.43@tcp [ 820.030588] Key type .llcrypt unregistered [ 820.031961] Key type ._llcrypt unregistered [ 822.793674] Key type ._llcrypt registered [ 822.795615] Key type .llcrypt registered [ 822.912325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.921084] alg: No test for adler32 (adler32-zlib) [ 823.738750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.772122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 823.785781] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 824.124996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 824.133130] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 824.501846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.43@tcp1 --nid 192.168.202.43@tcp2 [ 826.526162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp1 192.168.202.43@tcp2 [ 826.536918] LNetError: 75547:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.43@tcp1 to 192.168.202.43@tcp2 - Route table may be misconfigured [ 826.549353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp2 192.168.202.43@tcp1 [ 829.578736] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:56:51 (1786114611) [ 829.610312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.624661] LNet: 76113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.633233] LNetError: 76113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.637233] LNet: Removed LNI 192.168.202.43@tcp1 [ 830.694339] LNet: Removed LNI 192.168.202.43@tcp2 [ 830.712746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.750337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.762303] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 831.113796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 831.124539] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 831.538539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 831.567925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.43@tcp [ 831.589451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp1 192.168.202.43@tcp1 [ 831.614165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.43@tcp -d 192.168.202.43@tcp -e network_timeout [ 831.642204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -e network_timeout [ 831.663508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -e network_timeout [ 831.680402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -e network_timeout [ 831.698848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 831.709909] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 831.722597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 831.775728] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 831.780143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 831.822466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 834.971819] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:56:56 (1786114616) [ 835.042054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.054590] LNet: 76973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.062539] LNetError: 76973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.071399] LNetError: 76973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 835.075739] LNet: Removed LNI 192.168.202.43@tcp [ 836.133866] LNet: Removed LNI 192.168.202.43@tcp1 [ 836.149546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.195465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 836.310141] LNet: 77018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.502865] Key type .llcrypt unregistered [ 836.504680] Key type ._llcrypt unregistered [ 840.810270] Key type ._llcrypt registered [ 840.811542] Key type .llcrypt registered [ 841.398993] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:57:02 (1786114622) [ 841.598149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.610182] alg: No test for adler32 (adler32-zlib) [ 842.440070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.116737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 843.835903] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:57:05 (1786114625) [ 843.919583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.929406] LNet: 78114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.940878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.966713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.986461] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 844.346782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 844.362146] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 844.829268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 844.861917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp1 [ 844.886723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp1 [ 847.919346] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:57:09 (1786114629) [ 848.846326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg243-server [ 849.592362] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:57:11 (1786114631) [ 850.436906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg243-server [ 851.224239] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:57:12 (1786114632) [ 852.002183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg243-server [ 852.627474] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:57:14 (1786114634) [ 853.308041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg243-server [ 853.961252] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:57:15 (1786114635) [ 854.637992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg243-server [ 855.338324] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:57:16 (1786114636) [ 856.090395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg243-server [ 856.786980] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:57:18 (1786114638) [ 857.550601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg243-server [ 858.174443] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:57:19 (1786114639) [ 858.863065] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg243-server [ 859.482972] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:57:21 (1786114641) [ 860.272584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg243-server [ 860.969974] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:57:22 (1786114642) [ 861.004306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.014417] LNet: 81982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.017457] LNetError: 81982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.019610] LNet: Removed LNI 192.168.202.43@tcp [ 862.053763] LNet: Removed LNI 192.168.202.43@tcp1 [ 862.066554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.094487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.107112] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 862.415849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 862.452471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 862.521945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.530546] LNet: 82134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.534322] LNetError: 82134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.537957] LNetError: 82134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.545425] LNet: Removed LNI 192.168.202.43@tcp [ 862.554744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.578064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 862.593139] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 862.970591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 863.075346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.104320] LNet: Removed LNI 192.168.202.43@tcp [ 864.113768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.138515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.148951] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 864.423807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 864.456434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 864.516867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.527084] LNetError: 82426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.530245] LNetError: 82426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.539479] LNet: Removed LNI 192.168.202.43@tcp [ 864.548994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.572442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 864.583817] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 864.897850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 866.007060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.015917] LNet: 82581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.018508] LNet: 82581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 867.046165] LNet: Removed LNI 192.168.202.43@tcp [ 867.064906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.096060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.115630] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 867.467610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 867.510917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 868.625337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.636623] LNetError: 82737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.639163] LNetError: 82737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 868.652085] LNet: Removed LNI 192.168.202.43@tcp [ 868.660879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.691397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 868.703178] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 868.986077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 870.142178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.154586] LNet: 82891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.159201] LNet: 82891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 871.205581] LNet: Removed LNI 192.168.202.43@tcp [ 871.232672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.263635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.271519] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 871.599371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 871.636085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 871.755044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.809042] LNet: Removed LNI 192.168.202.43@tcp [ 872.821281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.852419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 872.866629] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 873.161887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 873.251900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.260846] LNetError: 83184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.263428] LNetError: 83184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 873.270414] LNet: Removed LNI 192.168.202.43@tcp [ 873.278209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.302531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.313672] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 873.595888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 873.625827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 873.725297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.789430] LNet: Removed LNI 192.168.202.43@tcp [ 874.798673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.832491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 874.846781] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 875.208304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 875.306420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.326397] LNet: Removed LNI 192.168.202.43@tcp [ 876.340495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.366958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.377309] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 876.673389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 876.706489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 876.787516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.861326] LNet: Removed LNI 192.168.202.43@tcp [ 877.869922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.899200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 877.910728] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 878.254043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 878.370913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.383294] LNet: 83768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.386906] LNet: 83768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 879.461313] LNet: Removed LNI 192.168.202.43@tcp [ 879.469813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.497225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.510583] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 879.862936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 879.886740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.902291] LNet: Removed LNI 192.168.202.43@tcp [ 879.914954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.940683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.957478] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 880.305391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 880.338284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.354473] LNet: Removed LNI 192.168.202.43@tcp [ 880.365569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.394746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 880.434704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.457457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.487304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 880.529820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.546568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.569383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 880.605767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.622856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.656731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 880.671088] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 881.019692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.037350] LNet: Removed LNI 192.168.202.43@tcp [ 881.045871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.092264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 884.241828] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:57:45 (1786114665) [ 884.303916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.329916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.358430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 884.370363] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/180] [ 884.889936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.903440] LNetError: 84926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.906127] LNetError: 84926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 884.915494] LNet: Removed LNI 192.168.202.43@tcp231 [ 884.924558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.954947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 884.965959] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/180] [ 884.968402] LNet: 84934:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.129248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 885.153212] LNet: Removed LNI 192.168.202.43@tcp231 [ 885.165742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 885.177173] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/180] [ 885.687122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.705073] LNet: Removed LNI 192.168.202.43@tcp231 [ 885.716235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.749513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 885.762161] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/0] [ 885.765100] LNet: 85086:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.940405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 885.962605] LNet: Removed LNI 192.168.202.43@tcp231 [ 885.972423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 885.985918] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/0] [ 886.489873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.509523] LNet: Removed LNI 192.168.202.43@tcp231 [ 886.522353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.563538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 886.575847] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/60] [ 886.738085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 886.757473] LNet: Removed LNI 192.168.202.43@tcp231 [ 886.769540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 886.785344] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/60] [ 887.276172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.298584] LNet: Removed LNI 192.168.202.43@tcp231 [ 887.307734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.336677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 887.350826] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/180] [ 887.353347] LNet: 85389:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 887.356411] LNet: 85389:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 887.517502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 887.538382] LNet: Removed LNI 192.168.202.43@tcp231 [ 887.547419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 887.557778] LNet: Added LNI 192.168.202.43@tcp231 [8/256/0/180] [ 890.608592] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:57:52 (1786114672) [ 890.656557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.673782] LNet: Removed LNI 192.168.202.43@tcp231 [ 890.682705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.711096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 890.730213] LNet: Added LNI 192.168.202.43@tcp232 [8/256/0/180] [ 891.052916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 891.071775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 891.088541] LNet: Removed LNI 192.168.202.43@tcp232 [ 891.096498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 891.106987] LNet: Added LNI 192.168.202.43@tcp232 [8/256/0/180] [ 891.388704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 891.405672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 891.428513] LNet: Removed LNI 192.168.202.43@tcp232 [ 891.436600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 891.450087] LNet: Added LNI 192.168.202.43@tcp232 [8/256/0/180] [ 891.784141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 891.803691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 891.827850] LNet: Removed LNI 192.168.202.43@tcp232 [ 891.836440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 891.850390] LNet: Added LNI 192.168.202.43@tcp232 [8/256/0/180] [ 892.198068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 892.218985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 892.236814] LNet: Removed LNI 192.168.202.43@tcp232 [ 892.245493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 894.588894] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:57:56 (1786114676) [ 894.781313] LNet: 87225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.783777] LNet: 87225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 895.048572] Key type .llcrypt unregistered [ 895.050046] Key type ._llcrypt unregistered [ 897.322698] Key type ._llcrypt registered [ 897.325028] Key type .llcrypt registered [ 897.412723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.419953] alg: No test for adler32 (adler32-zlib) [ 898.230332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 898.245754] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 901.694183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 905.060778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.076209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 905.088377] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 905.671328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_interrupt -r 2 [ 905.688180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_interrupt -r 2 [ 905.708139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_interrupt -r 2 [ 905.724843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_interrupt -r 2 [ 905.740499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_interrupt -r 2 [ 905.757815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_interrupt -r 2 [ 905.777193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_interrupt -r 2 [ 905.792988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_interrupt -r 2 [ 905.813172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 905.823896] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 905.827978] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 905.934775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 905.949138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 905.967839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 905.986212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 906.002714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 906.020337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 906.036097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 906.049813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 906.059817] LNet: 87884:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 906.071016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 906.086617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 906.103494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 906.119312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 906.134893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 906.151725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.362341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_dropped -r 2 [ 906.378926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_dropped -r 2 [ 906.403095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_dropped -r 2 [ 906.425178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_dropped -r 2 [ 906.443421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_dropped -r 2 [ 906.465438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_dropped -r 2 [ 906.487265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_dropped -r 2 [ 906.504555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_dropped -r 2 [ 906.522126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 906.534292] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 906.620557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 906.634892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 906.652835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 906.667116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 906.687568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 906.703943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 906.718632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 906.734244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 906.747836] LNet: 88005:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 906.758363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 906.775439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 906.795405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 906.812723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 906.828850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 906.846901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 907.034789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_aborted -r 2 [ 907.051564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_aborted -r 2 [ 907.067351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_aborted -r 2 [ 907.082884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_aborted -r 2 [ 907.104110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_aborted -r 2 [ 907.121246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_aborted -r 2 [ 907.139261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_aborted -r 2 [ 907.156315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_aborted -r 2 [ 907.176126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 907.291238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 907.306791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 907.322740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 907.338388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 907.354796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 907.369343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 907.388581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 907.403207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 907.424396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 907.438909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 907.452193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 907.465909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 907.480530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 907.494213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 907.693059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_no_route -r 2 [ 907.710768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_no_route -r 2 [ 907.729121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_no_route -r 2 [ 907.746695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_no_route -r 2 [ 907.763174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_no_route -r 2 [ 907.780660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_no_route -r 2 [ 907.797361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_no_route -r 2 [ 907.813847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_no_route -r 2 [ 907.829658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 907.841039] LNet: There was an unexpected network error while writing to 192.168.202.143: rc = -22 [ 907.843440] LNet: Skipped 3 previous similar messages [ 907.929618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 907.944609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 907.961331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 907.976470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 907.993988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 908.010330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 908.024594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 908.043478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 908.050963] LNet: 88245:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 908.054138] LNet: 88245:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 908.062149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 908.078860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 908.091405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 908.105986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 908.119600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.138259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.350277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e local_timeout -r 2 [ 908.365415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e local_timeout -r 2 [ 908.383534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e local_timeout -r 2 [ 908.401122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e local_timeout -r 2 [ 908.420538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e local_timeout -r 2 [ 908.440183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e local_timeout -r 2 [ 908.459631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e local_timeout -r 2 [ 908.476498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e local_timeout -r 2 [ 908.490619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 908.599532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 908.613608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 908.628725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 908.645842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 908.663924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 908.678965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 908.693608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 908.710743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 908.724106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 908.736892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 908.755190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 908.778122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 908.793525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.813589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.053911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp -m GET -e remote_dropped -r 2 [ 909.074716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.43@tcp -d 192.168.202.143@tcp1 -m GET -e remote_dropped -r 2 [ 909.094969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp -m GET -e remote_dropped -r 2 [ 909.112784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp -d 192.168.202.43@tcp1 -m GET -e remote_dropped -r 2 [ 909.137283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp -m GET -e remote_dropped -r 2 [ 909.158844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.143@tcp1 -m GET -e remote_dropped -r 2 [ 909.178129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp -m GET -e remote_dropped -r 2 [ 909.196529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.43@tcp1 -d 192.168.202.43@tcp1 -m GET -e remote_dropped -r 2 [ 909.217835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.43@tcp 192.168.202.143@tcp [ 909.231227] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.143@tcp [ 909.333524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 909.350182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 909.364745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 909.380200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 909.398612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 909.415181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 909.429388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 909.448614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 909.468652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 909.484338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 909.499431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 909.515036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 909.530512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 909.544514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 911.929365] LNet: 88558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.933421] LNetError: 88558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.936499] LNet: Removed LNI 192.168.202.43@tcp [ 911.946562] LNet: Removed LNI 192.168.202.43@tcp1 [ 912.106611] Key type .llcrypt unregistered [ 912.108247] Key type ._llcrypt unregistered [ 919.039158] Key type ._llcrypt registered [ 919.040600] Key type .llcrypt registered [ 919.558081] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:58:21 (1786114701) [ 919.730225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 919.735292] alg: No test for adler32 (adler32-zlib) [ 920.566747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.597014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 920.613614] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 920.992345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 921.011791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.43@tcp [ 921.023967] LNetError: 89537:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.43@tcp to recovery [ 921.071378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.43@tcp [ 921.124412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 923.745932] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:58:25 (1786114705) [ 923.823137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.834468] LNet: 90125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.836957] LNetError: 90125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.857747] LNet: Removed LNI 192.168.202.43@tcp [ 923.867485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.890077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.901399] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 924.166078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 924.177116] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 924.577725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.43@tcp [ 924.590118] LNetError: 90391:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.202.43@tcp to recovery [ 924.895351] LNet: 1 local NIs in recovery (showing 1): 192.168.202.43@tcp [ 924.898669] LNet: There was an unexpected network error while writing to 192.168.202.43: rc = -22 [ 925.682448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.43@tcp [ 930.146629] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:58:31 (1786114711) [ 930.824502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg243-server [ 931.448040] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:58:33 (1786114713) [ 932.136565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg243-server [ 932.745532] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:58:34 (1786114714) [ 933.562399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg243-server [ 934.276996] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:58:35 (1786114715) [ 934.317482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.329951] LNet: 92014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.334810] LNetError: 92014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.338811] LNet: Removed LNI 192.168.202.43@tcp [ 934.347768] LNet: Removed LNI 192.168.202.43@tcp1 [ 934.358818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.387090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.402863] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 934.724575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 934.757701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 934.781797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 934.814071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 934.834356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 934.854380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 937.177639] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:58:38 (1786114718) [ 937.216664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.226832] LNet: 92740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.231347] LNetError: 92740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.233853] LNetError: 92740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 937.243730] LNet: Removed LNI 192.168.202.43@tcp [ 937.256439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.280874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.292910] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 940.040274] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:58:41 (1786114721) [ 940.070706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.080217] LNet: 93441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.084169] LNetError: 93441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.093440] LNet: Removed LNI 192.168.202.43@tcp [ 940.103426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.124663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.133115] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 942.705900] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:58:44 (1786114724) [ 943.242093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 943.844412] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:58:45 (1786114725) [ 944.034765] LNet: Removed LNI 192.168.202.43@tcp [ 944.333154] Key type .llcrypt unregistered [ 944.334805] Key type ._llcrypt unregistered [ 946.528866] Key type ._llcrypt registered [ 946.530019] Key type .llcrypt registered [ 946.610869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.617584] alg: No test for adler32 (adler32-zlib) [ 947.432052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 947.444373] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 950.851339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 952.630164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.143@tcp [ 952.773174] LNet: 94896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.775291] LNetError: 94896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.786393] LNet: Removed LNI 192.168.202.43@tcp [ 952.982608] Key type .llcrypt unregistered [ 952.987039] Key type ._llcrypt unregistered [ 957.110293] Key type ._llcrypt registered [ 957.111592] Key type .llcrypt registered [ 957.605819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 958.222744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 958.986560] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:59:00 (1786114740) [ 959.673115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 960.369769] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:59:02 (1786114742) [ 961.125288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg243-server [ 961.820430] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:59:03 (1786114743) [ 962.578148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg243-server [ 963.203868] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:59:04 (1786114744) [ 963.801100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.808767] alg: No test for adler32 (adler32-zlib) [ 964.698638] Lustre: Lustre: Build Version: 2.17.55_1_gd52575f [ 964.805594] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 966.399153] Key type lgssc registered [ 966.806362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.256654] Key type lgssc unregistered [ 975.389582] LNet: 98065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.392752] LNetError: 98065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.405528] LNet: Removed LNI 192.168.202.43@tcp [ 975.672444] Key type .llcrypt unregistered [ 975.674232] Key type ._llcrypt unregistered [ 981.018330] Key type ._llcrypt registered [ 981.019733] Key type .llcrypt registered [ 981.558633] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:59:23 (1786114763) [ 981.737309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 981.742913] alg: No test for adler32 (adler32-zlib) [ 982.556656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.584267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 982.606550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 982.628027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 982.648376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 985.162581] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:59:26 (1786114766) [ 985.202384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.210647] LNet: 99472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.221558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.248865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.258662] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 985.602562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.611415] LNetError: 99621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.619269] LNet: Removed LNI 192.168.202.43@tcp [ 985.627652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.650669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 985.662170] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 985.839494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.850189] LNet: 99649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.853366] LNet: 99649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 985.866312] LNet: Removed LNI 192.168.202.43@tcp [ 985.876591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.900895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 985.913155] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 988.332150] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:59:30 (1786114770) [ 988.368580] LNet: 100234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.371554] LNetError: 100234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.374966] LNetError: 100234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 988.385918] LNet: Removed LNI 192.168.202.43@tcp [ 988.633477] Key type .llcrypt unregistered [ 988.634676] Key type ._llcrypt unregistered [ 990.802726] Key type ._llcrypt registered [ 990.804240] Key type .llcrypt registered [ 990.879944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.888826] alg: No test for adler32 (adler32-zlib) [ 991.696439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.732340] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 991.733843] LNet: *** cfs_fail_loc=e003, val=100*** [ 992.003832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 992.013175] Ping reply received with mapping: [ 992.015478] found metadata reply NI[192.168.202.43@tcp] [ 992.018137] 2b ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 992.021132] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 992.023804] 5a 5a 5a 5a 5a 5a 5a 5a [ 992.084124] LNet: 100636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.087383] LNetError: 100636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.095598] LNet: Removed LNI 192.168.202.43@tcp [ 992.361458] Key type .llcrypt unregistered [ 992.363030] Key type ._llcrypt unregistered [ 996.509304] Key type ._llcrypt registered [ 996.510733] Key type .llcrypt registered [ 997.045292] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:59:38 (1786114778) [ 997.228383] Key type .llcrypt unregistered [ 997.229935] Key type ._llcrypt unregistered [ 999.380595] Key type ._llcrypt registered [ 999.383542] Key type .llcrypt registered [ 999.467425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.476068] alg: No test for adler32 (adler32-zlib) [ 1000.375722] LNetError: 101679:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1000.534042] Key type .llcrypt unregistered [ 1000.535406] Key type ._llcrypt unregistered [ 1002.543830] Key type ._llcrypt registered [ 1002.545107] Key type .llcrypt registered [ 1003.073281] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:59:44 (1786114784) [ 1003.264281] Key type .llcrypt unregistered [ 1003.266160] Key type ._llcrypt unregistered [ 1005.750238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1005.752562] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.967577] Key type ._llcrypt registered [ 1007.968959] Key type .llcrypt registered [ 1008.037451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.044960] alg: No test for adler32 (adler32-zlib) [ 1008.859242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.885184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1013.145575] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:59:54 (1786114794) [ 1013.199087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.208343] LNet: 103455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.210783] LNetError: 103455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1013.218239] LNet: Removed LNI 10.1.2.3@tcp [ 1013.227886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.514564] Key type .llcrypt unregistered [ 1013.516308] Key type ._llcrypt unregistered [ 1015.736253] Key type ._llcrypt registered [ 1015.738559] Key type .llcrypt registered [ 1015.817210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.824791] alg: No test for adler32 (adler32-zlib) [ 1016.655423] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1016.688377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 1016.756811] LNet: 103745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.766593] LNetError: 103745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.782637] LNet: Removed LNI 192.168.202.43@tcp [ 1017.089361] Key type .llcrypt unregistered [ 1017.091253] Key type ._llcrypt unregistered [ 1021.319443] Key type ._llcrypt registered [ 1021.321191] Key type .llcrypt registered [ 1021.894026] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:00:03 (1786114803) [ 1022.134797] Key type .llcrypt unregistered [ 1022.136112] Key type ._llcrypt unregistered [ 1024.319042] Key type ._llcrypt registered [ 1024.320603] Key type .llcrypt registered [ 1024.413395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.422724] alg: No test for adler32 (adler32-zlib) [ 1027.856169] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:00:09 (1786114809) [ 1027.898595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.915639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.943603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.953497] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1028.268380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.286489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1028.306436] LNet: Removed LNI 192.168.202.43@tcp [ 1028.393332] LNet: 105486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.562799] Key type .llcrypt unregistered [ 1028.564093] Key type ._llcrypt unregistered [ 1032.997498] Key type ._llcrypt registered [ 1032.998773] Key type .llcrypt registered [ 1033.606716] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:00:15 (1786114815) [ 1033.794165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.801690] alg: No test for adler32 (adler32-zlib) [ 1034.620812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.650093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.663127] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1034.995677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 1035.013917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.43@tcp [ 1035.024380] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1035.026191] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1035.028478] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1035, 0, 0 [ 1035.030609] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1035.032348] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1035.034693] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1035.037032] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1035, 0, 0 [ 1035.040467] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1035.042276] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1035.044026] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1035.046085] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1035, 0, 1085 [ 1035.048713] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1035.050073] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1035.052020] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1035.053619] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1035, 0, 1085 [ 1035.056583] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1037.561481] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:00:19 (1786114819) [ 1037.726501] LNet: 107024:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.728976] LNetError: 107024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.742628] LNet: Removed LNI 192.168.202.43@tcp [ 1038.008615] Key type .llcrypt unregistered [ 1038.010079] Key type ._llcrypt unregistered [ 1040.279728] Key type ._llcrypt registered [ 1040.281594] Key type .llcrypt registered [ 1040.401485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.408500] alg: No test for adler32 (adler32-zlib) [ 1041.236181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1041.252718] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1044.592151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 1048.061702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.076290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1048.084770] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 1048.465404] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1048.521495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1048.523773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1048.526184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1050.674471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1050.683911] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1054.665818] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:00:36 (1786114836) [ 1054.713075] LNet: 108525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.719072] LNetError: 108525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.723563] LNet: Removed LNI 192.168.202.43@tcp [ 1055.775145] LNet: Removed LNI 192.168.202.43@tcp1 [ 1055.776918] LNetError: 108525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.779770] LNetError: 108525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1055.789734] LNet: Removed LNI 10.1.2.3@tcp99 [ 1056.050631] Key type .llcrypt unregistered [ 1056.052256] Key type ._llcrypt unregistered [ 1058.363302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1058.365836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1060.569743] Key type ._llcrypt registered [ 1060.571318] Key type .llcrypt registered [ 1060.647152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.652446] alg: No test for adler32 (adler32-zlib) [ 1061.461781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.482966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1061.491451] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1061.776198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1061.787616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1062.654687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1062.672897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1062.693853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1062.703725] LNetError: 109273:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1062.719634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1062.738689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1062.909642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1062.940651] LNet: 109293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.945284] LNetError: 109293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1062.948739] LNet: Removed LNI 10.1.2.3@tcp [ 1063.973341] LNet: Removed LNI 192.168.202.43@tcp [ 1064.239584] Key type .llcrypt unregistered [ 1064.241192] Key type ._llcrypt unregistered [ 1068.270851] Key type ._llcrypt registered [ 1068.272010] Key type .llcrypt registered [ 1068.799828] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:00:50 (1786114850) [ 1068.966382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1068.972881] alg: No test for adler32 (adler32-zlib) [ 1069.781703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.805644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1069.816951] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1070.164385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.43@tcp [ 1070.196538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg243-client@tcp [ 1072.710985] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:00:54 (1786114854) [ 1072.761845] LNet: 110821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.766929] LNetError: 110821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.774392] LNet: Removed LNI 192.168.202.43@tcp [ 1073.021425] Key type .llcrypt unregistered [ 1073.022662] Key type ._llcrypt unregistered [ 1075.361850] Key type ._llcrypt registered [ 1075.363030] Key type .llcrypt registered [ 1075.435260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1075.442355] alg: No test for adler32 (adler32-zlib) [ 1076.254978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1076.265636] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1076.278697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1079.135906] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:01:00 (1786114860) [ 1079.790599] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1080.619496] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:01:02 (1786114862) [ 1081.314944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1081.993392] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:01:03 (1786114863) [ 1082.040360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.052039] LNet: 112250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.055516] LNetError: 112250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.062449] LNet: Removed LNI 192.168.202.43@tcp [ 1082.074281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.104888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1082.117292] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1082.434614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1082.459953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1085.272631] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:01:06 (1786114866) [ 1085.447906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.460624] LNet: 112965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.465442] LNetError: 112965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.482550] LNet: Removed LNI 192.168.202.43@tcp [ 1085.491708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.515538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1085.530282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1085.792296] Key type .llcrypt unregistered [ 1085.793576] Key type ._llcrypt unregistered [ 1090.370726] Key type ._llcrypt registered [ 1090.372364] Key type .llcrypt registered [ 1091.015266] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:01:12 (1786114872) [ 1091.212199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1091.218427] alg: No test for adler32 (adler32-zlib) [ 1092.053652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.083776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1092.098583] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1092.431484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1092.460162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1095.241233] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:01:16 (1786114876) [ 1095.301795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.313986] LNet: 114517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.319384] LNetError: 114517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.335022] LNet: Removed LNI 192.168.202.43@tcp [ 1095.347550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.384343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1095.399451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1098.065379] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:01:19 (1786114879) [ 1098.093521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.103228] LNet: 115091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.114026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.147888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1098.300107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1098.315081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1098.342760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1098.496813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1098.515314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1098.547690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.563188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.590060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1098.736329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1098.750439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1098.777503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1098.926854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1098.940320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1101.633623] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:01:23 (1786114883) [ 1101.666872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1101.677704] LNet: 115725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.680457] LNet: 115725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1101.692973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.734486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1101.751194] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1102.105533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1102.116112] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1102.521517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1102.547442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1102.661617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1102.675048] LNetError: 116010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.678376] LNet: Removed LNI 192.168.202.43@tcp [ 1102.684642] LNet: Removed LNI 192.168.202.43@tcp2 [ 1102.695677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.721902] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1102.724801] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1102.727172] LNet: 116016:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.923623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1103.009424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.031149] LNet: Removed LNI 192.168.202.43@tcp [ 1104.033127] LNetError: 116053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.035268] LNetError: 116053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1104.043461] LNet: Removed LNI 192.168.202.43@tcp2 [ 1104.053629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.077638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1104.088561] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1104.374524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1104.383886] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1104.777728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1104.800525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1104.828771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1104.957981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.967423] LNet: 116337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.970140] LNet: 116337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1106.015180] LNet: Removed LNI 192.168.202.43@tcp [ 1107.047407] LNet: Removed LNI 192.168.202.43@tcp2 [ 1107.071691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.107805] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1107.112376] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1107.115115] LNet: 116343:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.356493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1107.475584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1107.488154] LNetError: 116380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1107.491464] LNetError: 116380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1107.494554] LNet: Removed LNI 192.168.202.43@tcp [ 1108.519140] LNet: Removed LNI 192.168.202.43@tcp2 [ 1108.528671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.553581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1108.566573] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1108.889989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1108.903680] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1109.277063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1109.303538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1109.397326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1109.408360] LNet: 116660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.410975] LNet: 116660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1110.431119] LNet: Removed LNI 192.168.202.43@tcp [ 1110.438352] LNet: Removed LNI 192.168.202.43@tcp2 [ 1110.448877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.480322] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1110.482923] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1110.485434] LNet: 116666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.683838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1110.766812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1111.839246] LNet: Removed LNI 192.168.202.43@tcp [ 1111.841097] LNetError: 116703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.843881] LNetError: 116703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1111.853310] LNet: Removed LNI 192.168.202.43@tcp2 [ 1111.863408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.886559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1111.898583] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1112.240938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1112.250607] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1112.691115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1112.708576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1112.736757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1112.825192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1113.887165] LNet: Removed LNI 192.168.202.43@tcp [ 1114.917812] LNet: Removed LNI 192.168.202.43@tcp2 [ 1114.929303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.962053] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1114.966270] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1114.969922] LNet: 116993:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.163544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1115.246815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1115.255873] LNet: Removed LNI 192.168.202.43@tcp [ 1116.326379] LNet: Removed LNI 192.168.202.43@tcp2 [ 1116.337118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.359281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1116.373185] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1116.699251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1116.708336] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1117.092339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1117.107601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1117.142400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1117.291822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1118.367174] LNet: Removed LNI 192.168.202.43@tcp [ 1118.375487] LNet: Removed LNI 192.168.202.43@tcp2 [ 1118.383358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1118.411722] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1118.414897] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1118.602569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1118.746749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1118.758610] LNet: 117385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.761894] LNet: 117385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1119.777452] LNet: Removed LNI 192.168.202.43@tcp [ 1119.785913] LNet: Removed LNI 192.168.202.43@tcp2 [ 1119.802522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.833238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1119.851705] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1120.168910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1120.179918] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1120.559673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1120.579276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1120.595142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1120.623382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1120.794377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1120.804157] LNetError: 117686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.807868] LNetError: 117686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1120.810956] LNet: Removed LNI 192.168.202.43@tcp [ 1121.830758] LNet: Removed LNI 192.168.202.43@tcp2 [ 1121.839607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.867853] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1121.871388] LNet: Added LNI 192.168.202.43@tcp2 [8/256/0/180] [ 1121.873864] LNet: 117692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.876037] LNet: 117692:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1122.055279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1124.823602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1125.450231] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:01:47 (1786114907) [ 1125.484867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.496218] LNet: Removed LNI 192.168.202.43@tcp [ 1126.565534] LNet: Removed LNI 192.168.202.43@tcp2 [ 1126.575190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.683772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1126.697646] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1126.992904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1127.002451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.353411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.364566] LNet: Removed LNI 10.1.2.3@tcp [ 1127.376625] LNet: Removed LNI 192.168.202.43@tcp [ 1127.389476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.417137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1127.429637] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1127.431674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.601511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1127.613621] LNet: Removed LNI 192.168.202.43@tcp [ 1127.628613] LNet: Removed LNI 10.1.2.3@tcp [ 1130.162041] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:01:51 (1786114911) [ 1130.248103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1130.265236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.306895] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1130.377196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1130.379737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1130.382168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1132.499584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.511384] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1132.847678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1132.857087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1133.684160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.43@tcp [ 1133.703425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1133.730571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.43@tcp -m PUT -e local_timeout -r 1 [ 1133.767097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1133.797804] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1133.799928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1133.914909] LNetError: 119389:0:(lib-move.c:4771:lnet_parse()) 192.168.202.43@tcp, src 192.168.202.43@tcp: Dropping PUT (error -108 looking up sender) [ 1134.943420] LNet: 119847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.946500] LNet: 119847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1134.949816] LNet: Removed LNI 10.1.2.3@tcp [ 1135.977355] LNet: Removed LNI 192.168.202.43@tcp [ 1136.142040] Key type .llcrypt unregistered [ 1136.144102] Key type ._llcrypt unregistered [ 1140.755044] Key type ._llcrypt registered [ 1140.756277] Key type .llcrypt registered [ 1141.342056] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:02:02 (1786114922) [ 1141.506386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1141.514088] alg: No test for adler32 (adler32-zlib) [ 1142.343134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.498953] LNet: 120698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.775425] Key type .llcrypt unregistered [ 1142.776845] Key type ._llcrypt unregistered [ 1145.262676] Key type ._llcrypt registered [ 1145.264147] Key type .llcrypt registered [ 1145.337743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1145.343718] alg: No test for adler32 (adler32-zlib) [ 1146.165702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1146.181971] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1149.686569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 1149.764662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1150.050241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.143@tcp [ 1150.152769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.165270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.180065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.194992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.216200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.232966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.249621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.263294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.272329] LNet: 121147:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.280950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.296442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.303085] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.143@tcp [ 1150.311244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.326131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.341663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.360124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.635553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1150.728675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.744621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.759832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.773923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.789722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.806147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.820537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.834041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.842814] LNet: 121264:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.855619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1150.875501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1150.890839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1150.911174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.934545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.954699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.269909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.143@tcp [ 1151.368846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.385333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.399900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.416484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.431369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.446773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.462741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.483360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.500392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.516996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.531346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.547475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.565429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.580695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.828536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1151.917614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.933346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.950450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.964525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.978970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.995909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.010607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.026258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.035780] LNet: 121498:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.038907] LNet: 121498:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1152.047755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.066116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.083877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.098211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.113872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.129936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.258403] LNet: 121544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.261440] LNetError: 121544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.269411] LNet: Removed LNI 192.168.202.43@tcp [ 1152.435904] Key type .llcrypt unregistered [ 1152.437413] Key type ._llcrypt unregistered [ 1158.767828] Key type ._llcrypt registered [ 1158.769022] Key type .llcrypt registered [ 1159.356039] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:02:20 (1786114940) [ 1159.536185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.541491] alg: No test for adler32 (adler32-zlib) [ 1160.347688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.513518] LNet: 122428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.787497] Key type .llcrypt unregistered [ 1160.789280] Key type ._llcrypt unregistered [ 1163.144803] Key type ._llcrypt registered [ 1163.145814] Key type .llcrypt registered [ 1163.241549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.248660] alg: No test for adler32 (adler32-zlib) [ 1164.066703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1164.080263] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1167.472732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.143@tcp [ 1171.254733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.279815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1171.294484] LNet: Added LNI 192.168.202.43@tcp1 [8/256/0/180] [ 1171.699841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1171.945653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.143@tcp [ 1172.042339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.056885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.072945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.087659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.102352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.118842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.134034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.149546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.157770] LNet: 123078:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.167161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.188331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.203541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.222324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.238192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.255455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.548939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1172.642705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.661387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.675118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.691461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.707724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.723788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.738252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.752411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.761869] LNet: 123195:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.772123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.792657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.808614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.825731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.846255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.864281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.119697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.143@tcp [ 1173.214335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.229422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.244664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.260013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.276846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.281231] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.143@tcp [ 1173.292151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.309303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.324132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.339381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.356445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.377213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.392991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.421860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.443957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.741469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1173.883409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.906723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.924305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.944418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.970996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.991696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.015216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.036424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.046344] LNet: 123429:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.049215] LNet: 123429:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1174.059681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.074809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.089407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.110416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.123946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.140167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.621891] LNet: 123510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.625592] LNetError: 123510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1176.628612] LNet: Removed LNI 192.168.202.43@tcp [ 1176.645800] LNet: Removed LNI 192.168.202.43@tcp1 [ 1176.825715] Key type .llcrypt unregistered [ 1176.827081] Key type ._llcrypt unregistered [ 1183.554781] Key type ._llcrypt registered [ 1183.555918] Key type .llcrypt registered [ 1184.156202] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:02:45 (1786114965) [ 1184.807775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1185.514930] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:02:47 (1786114967) [ 1188.558412] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:02:50 (1786114970) [ 1191.794321] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:02:53 (1786114973) [ 1194.757287] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:02:56 (1786114976) [ 1197.673922] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:02:59 (1786114979) [ 1200.906257] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:03:02 (1786114982) [ 1203.908065] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:03:05 (1786114985) [ 1207.242112] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:03:08 (1786114988) [ 1210.554680] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:03:12 (1786114992) [ 1213.520566] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:03:15 (1786114995) [ 1216.375857] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:03:17 (1786114997) [ 1219.219447] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:03:20 (1786115000) [ 1222.439402] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:03:23 (1786115003) [ 1225.560727] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:03:27 (1786115007) [ 1229.109335] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:03:30 (1786115010) [ 1232.148196] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:03:33 (1786115013) [ 1235.702929] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:03:37 (1786115017) [ 1238.794953] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:03:40 (1786115020) [ 1238.951982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.957798] alg: No test for adler32 (adler32-zlib) [ 1239.774968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.823299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1239.833808] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1240.322666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.330501] LNet: 134493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.332814] LNetError: 134493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.347771] LNet: Removed LNI 192.168.202.43@tcp [ 1240.364896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.410421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1240.423611] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1243.484454] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:03:45 (1786115025) [ 1246.450543] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:03:48 (1786115028) [ 1249.318468] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:03:50 (1786115030) [ 1252.265357] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:03:53 (1786115033) [ 1254.982800] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:03:56 (1786115036) [ 1257.572942] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:03:59 (1786115039) [ 1260.185619] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:04:01 (1786115041) [ 1262.837660] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:04:04 (1786115044) [ 1265.508689] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:04:07 (1786115047) [ 1268.032653] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:04:09 (1786115049) [ 1270.488364] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:04:12 (1786115052) [ 1273.112719] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:04:14 (1786115054) [ 1275.717572] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1113 sec ========== 11:04:17 (1786115057) [ 1275.832869] LNet: 141949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.836521] LNetError: 141949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.848484] LNet: Removed LNI 192.168.202.43@tcp [ 1276.129709] Key type .llcrypt unregistered [ 1276.131593] Key type ._llcrypt unregistered [ 1280.369742] Key type ._llcrypt registered [ 1280.371473] Key type .llcrypt registered [ 1280.548255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1280.553553] alg: No test for adler32 (adler32-zlib) [ 1281.427926] Lustre: Lustre: Build Version: 2.17.55_1_gd52575f [ 1281.531777] LNet: Added LNI 192.168.202.43@tcp [8/256/0/180] [ 1283.127148] Key type lgssc registered [ 1283.586353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1307.410584] Lustre: Mounted lustre-client [ 1309.842380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1312.310111] Lustre: Unmounted lustre-client [ 1324.685271] Key type lgssc unregistered [ 1324.812666] LNet: 144865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.818785] LNetError: 144865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.830949] LNet: Removed LNI 192.168.202.43@tcp [ 1325.117131] Key type .llcrypt unregistered [ 1325.118520] Key type ._llcrypt unregistered