[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 409801104 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002366] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010050] LSM: Security Framework initializing [ 0.011031] Yama: becoming mindful. [ 0.011719] SELinux: Initializing. [ 0.012062] *** VALIDATE selinux *** [ 0.019670] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023633] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025122] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026085] *** VALIDATE tmpfs *** [ 0.027319] *** VALIDATE proc *** [ 0.029057] *** VALIDATE cgroup *** [ 0.030000] *** VALIDATE cgroup2 *** [ 0.030290] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033021] Spectre V2 : User space: Vulnerable [ 0.033735] Speculative Store Bypass: Vulnerable [ 0.036033] debug: unmapping init [mem 0xffffffffa0859000-0xffffffffa0860fff] [ 0.039145] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042011] ... bit width: 48 [ 0.043010] ... generic registers: 4 [ 0.044008] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046012] ... fixed-purpose events: 3 [ 0.047010] ... event mask: 000000070000000f [ 0.048316] rcu: Hierarchical SRCU implementation. [ 0.050106] smp: Bringing up secondary CPUs ... [ 0.051553] x86: Booting SMP configuration: [ 0.052024] .... node #0, CPUs: #1 #2 #3 [ 0.055143] smp: Brought up 1 node, 4 CPUs [ 0.057016] smpboot: Max logical packages: 1 [ 0.058014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140072] node 0 deferred pages initialised in 79ms [ 0.144101] devtmpfs: initialized [ 0.145208] x86/mm: Memory block size: 128MB [ 0.147834] gcov: version magic: 0x41383552 [ 0.149302] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151278] pinctrl core: initialized pinctrl subsystem [ 0.152257] [ 0.152743] ************************************************************* [ 0.153027] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154019] ** ** [ 0.155018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156016] ** ** [ 0.157014] ** This means that this kernel is built to expose internal ** [ 0.158018] ** IOMMU data structures, which may compromise security on ** [ 0.159015] ** your system. ** [ 0.160011] ** ** [ 0.161012] ** If you see this message and you are not debugging the ** [ 0.162016] ** kernel, report this immediately to your vendor! ** [ 0.163015] ** ** [ 0.164014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165014] ************************************************************* [ 0.166680] NET: Registered protocol family 16 [ 0.167490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.170527] cpuidle: using governor menu [ 0.171000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.174582] PCI: Using configuration type 1 for base access [ 0.176126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190047] cryptd: max_cpu_qlen set to 1000 [ 0.192280] ACPI: Added _OSI(Module Device) [ 0.194011] ACPI: Added _OSI(Processor Device) [ 0.195015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.201340] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206399] ACPI: Interpreter enabled [ 0.207048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208010] ACPI: Using IOAPIC for interrupt routing [ 0.209080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210298] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218921] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221064] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.222982] acpiphp: Slot [2] registered [ 0.223099] acpiphp: Slot [5] registered [ 0.223891] acpiphp: Slot [6] registered [ 0.224093] acpiphp: Slot [3] registered [ 0.224856] acpiphp: Slot [4] registered [ 0.225105] acpiphp: Slot [7] registered [ 0.225800] acpiphp: Slot [8] registered [ 0.226053] acpiphp: Slot [9] registered [ 0.226906] acpiphp: Slot [10] registered [ 0.227061] acpiphp: Slot [11] registered [ 0.227826] acpiphp: Slot [12] registered [ 0.228081] acpiphp: Slot [13] registered [ 0.228898] acpiphp: Slot [14] registered [ 0.229053] acpiphp: Slot [15] registered [ 0.229719] acpiphp: Slot [16] registered [ 0.230074] acpiphp: Slot [17] registered [ 0.231088] acpiphp: Slot [18] registered [ 0.231924] acpiphp: Slot [19] registered [ 0.232053] acpiphp: Slot [20] registered [ 0.233082] acpiphp: Slot [21] registered [ 0.234129] acpiphp: Slot [22] registered [ 0.235081] acpiphp: Slot [23] registered [ 0.235956] acpiphp: Slot [24] registered [ 0.236060] acpiphp: Slot [25] registered [ 0.236719] acpiphp: Slot [26] registered [ 0.237049] acpiphp: Slot [27] registered [ 0.238107] acpiphp: Slot [28] registered [ 0.239097] acpiphp: Slot [29] registered [ 0.240111] acpiphp: Slot [30] registered [ 0.241107] acpiphp: Slot [31] registered [ 0.242057] PCI host bridge to bus 0000:00 [ 0.243022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.244023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.245025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.246023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.247027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.248041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.249316] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252081] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255543] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.263887] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.267009] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.283026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.285719] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.287597] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.290033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.293042] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.297016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.305965] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.315332] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.319015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.324013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.334015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.342104] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.348018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.353016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.367014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.375822] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.377303] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.380523] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.382537] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.385226] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.389104] iommu: Default domain type: Passthrough [ 0.391570] SCSI subsystem initialized [ 0.392078] ACPI: bus type USB registered [ 0.393076] usbcore: registered new interface driver usbfs [ 0.395067] usbcore: registered new interface driver hub [ 0.396051] usbcore: registered new device driver usb [ 0.397330] pps_core: LinuxPPS API ver. 1 registered [ 0.399008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.401060] PTP clock support registered [ 0.404160] EDAC MC: Ver: 3.0.0 [ 0.407165] PCI: Using ACPI for IRQ routing [ 0.408457] NetLabel: Initializing [ 0.409007] NetLabel: domain hash size = 128 [ 0.410008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.411081] NetLabel: unlabeled traffic allowed by default [ 0.413104] vgaarb: loaded [ 0.414257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.415008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.421378] clocksource: Switched to clocksource kvm-clock [ 0.531083] VFS: Disk quotas dquot_6.6.0 [ 0.532742] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.535128] *** VALIDATE ramfs *** [ 0.536182] *** VALIDATE hugetlbfs *** [ 0.537783] pnp: PnP ACPI init [ 0.539675] pnp: PnP ACPI: found 6 devices [ 0.567541] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.569786] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.571328] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.572845] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.574478] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.575845] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.577791] NET: Registered protocol family 2 [ 0.580152] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.584349] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.588978] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.596121] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.600586] TCP: Hash tables configured (established 65536 bind 65536) [ 0.604283] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.608416] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.611566] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.616443] NET: Registered protocol family 1 [ 0.618814] RPC: Registered named UNIX socket transport module. [ 0.620225] RPC: Registered udp transport module. [ 0.621367] RPC: Registered tcp transport module. [ 0.622452] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.624010] NET: Registered protocol family 44 [ 0.625030] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.626343] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.627705] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.629284] PCI: CLS 0 bytes, default 64 [ 0.630873] Unpacking initramfs... [ 1.988607] debug: unmapping init [mem 0xffff8e0afcc64000-0xffff8e0afffcffff] [ 1.992523] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.994695] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.997544] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.461240] Initialise system trusted keyrings [ 2.462980] Key type blacklist registered [ 2.464539] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.471782] zbud: loaded [ 2.474945] *** VALIDATE nfs *** [ 2.475854] *** VALIDATE nfs4 *** [ 2.477444] pstore: using deflate compression [ 2.480081] Platform Keyring initialized [ 2.561695] NET: Registered protocol family 38 [ 2.562794] Key type asymmetric registered [ 2.563742] Asymmetric key parser 'x509' registered [ 2.565130] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.567243] io scheduler mq-deadline registered [ 2.568373] io scheduler kyber registered [ 2.569353] io scheduler bfq registered [ 2.570416] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.572177] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.573869] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.575632] ACPI: Power Button [PWRF] [ 2.579641] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.584755] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.598352] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.623976] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.649302] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.652792] Non-volatile memory driver v1.3 [ 2.654263] Linux agpgart interface v0.103 [ 2.682020] virtio_blk virtio1: [vda] 134216 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.684134] vda: detected capacity change from 0 to 68718592 [ 2.697959] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.700604] vdb: detected capacity change from 0 to 1073741824 [ 2.706334] libphy: Fixed MDIO Bus: probed [ 2.712303] usbcore: registered new interface driver usbserial_generic [ 2.714614] usbserial: USB Serial support registered for generic [ 2.716655] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.721534] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.722907] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.724930] mousedev: PS/2 mouse device common for all mice [ 2.727682] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.729596] rtc_cmos 00:05: RTC can wake from S4 [ 2.733536] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.735907] rtc_cmos 00:05: registered as rtc0 [ 2.739515] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.739541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.741382] intel_pstate: CPU model not supported [ 2.746206] hid: raw HID events driver (C) Jiri Kosina [ 2.748063] usbcore: registered new interface driver usbhid [ 2.750116] usbhid: USB HID core driver [ 2.751702] drop_monitor: Initializing network drop monitor service [ 2.754119] Initializing XFRM netlink socket [ 2.756078] NET: Registered protocol family 10 [ 2.758838] Segment Routing with IPv6 [ 2.759697] NET: Registered protocol family 17 [ 2.760770] mpls_gso: MPLS GSO support [ 2.765795] RAS: Correctable Errors collector initialized. [ 2.767570] AVX version of gcm_enc/dec engaged. [ 2.769070] AES CTR mode by8 optimization enabled [ 2.842072] sched_clock: Marking stable (2841963477, 0)->(3650117239, -808153762) [ 2.845432] registered taskstats version 1 [ 2.847769] Loading compiled-in X.509 certificates [ 2.850648] zswap: loaded using pool lzo/zbud [ 2.874347] Key type big_key registered [ 2.886210] Key type encrypted registered [ 2.886951] ima: No TPM chip found, activating TPM-bypass! [ 2.888459] ima: Allocated hash algorithm: sha1 [ 2.889483] ima: No architecture policies found [ 2.890487] evm: Initialising EVM extended attributes: [ 2.891572] evm: security.selinux [ 2.892368] evm: security.ima [ 2.892901] evm: security.capability [ 2.893655] evm: HMAC attrs: 0x1 [ 2.895319] rtc_cmos 00:05: setting system clock to 2026-01-30 05:47:06 UTC (1769752026) [ 2.899387] debug: unmapping init [mem 0xffffffffa1803000-0xffffffffa19fffff] [ 2.901297] debug: unmapping init [mem 0xffffffffa0582000-0xffffffffa0858fff] [ 2.908092] Write protecting the kernel read-only data: 28672k [ 2.910168] debug: unmapping init [mem 0xffffffff9ec03000-0xffffffff9edfffff] [ 2.911713] debug: unmapping init [mem 0xffffffff9f514000-0xffffffff9f5fffff] [ 2.936267] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.942035] systemd[1]: Detected virtualization kvm. [ 2.943386] systemd[1]: Detected architecture x86-64. [ 2.945120] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.973620] systemd[1]: No hostname configured. [ 2.975057] systemd[1]: Set hostname to . [ 2.976933] random: systemd: uninitialized urandom read (16 bytes read) [ 2.978885] systemd[1]: Initializing machine ID from random generator. [ 3.010986] random: ln: uninitialized urandom read (6 bytes read) [ 3.114442] random: systemd: uninitialized urandom read (16 bytes read) [ 3.117352] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.121991] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.126290] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.771711] device-mapper: uevent: version 1.0.3 [ 3.773947] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.435861] virtio_net virtio0 ens2: renamed from eth0 [ 4.499835] scsi host0: ata_piix [ 4.502487] scsi host1: ata_piix [ 4.520481] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.523942] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.788447] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.398466] random: crng init done [ 9.399883] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.750321] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.892466] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.127511] SELinux: Disabled at runtime. [ 11.183672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.192656] systemd[1]: Detected virtualization kvm. [ 11.194389] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.623608] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.627093] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.630923] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.633748] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.636599] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.645771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.649471] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ 11.787893] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.074216] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.368681] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.515678] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.617128] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.638114] EDAC sbridge: Ver: 1.1.2 [ 13.574692] Key type dns_resolver registered [ 13.888907] NFS: Registering the id_resolver key type [ 13.891962] Key type id_resolver registered [ 13.894374] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg142-client login: [ 39.852186] libcfs: loading out-of-tree module taints kernel. [ 40.013490] Key type ._llcrypt registered [ 40.018228] Key type .llcrypt registered [ 40.917350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.941810] alg: No test for adler32 (adler32-zlib) [ 42.562732] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 43.554844] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 45.303162] Key type lgssc registered [ 47.705820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 128.701642] hrtimer: interrupt took 6624507 ns [ 168.049338] Lustre: Mounted lustre-client [ 173.459530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 193.508420] Lustre: lustre-OST0000-osc-ffff8e0b52d2f000: disconnect after 23s idle [ 196.001977] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing check_logdir /tmp/testlogs/ [ 200.689405] Lustre: DEBUG MARKER: oleg142-client.virtnet: executing yml_node [ 204.686950] Lustre: DEBUG MARKER: Client: 2.17.50.72 [ 207.355654] Lustre: DEBUG MARKER: MDS: 2.17.50.72 [ 209.010625] Lustre: DEBUG MARKER: OSS: 2.17.50.72 [ 211.197294] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jan 30 00:50:33 EST 2026 [ 229.476193] LustreError: 7921:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e0b52d2f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 229.484540] LustreError: 7921:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 229.532469] Lustre: Unmounted lustre-client [ 252.196442] Key type lgssc unregistered [ 252.394636] LNet: 8367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.402914] LNetError: 8367:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 252.418645] LNet: Removed LNI 192.168.201.42@tcp [ 253.262280] Key type .llcrypt unregistered [ 253.266262] Key type ._llcrypt unregistered [ 284.456833] Key type ._llcrypt registered [ 284.465897] Key type .llcrypt registered [ 284.715585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 284.744596] alg: No test for adler32 (adler32-zlib) [ 285.708700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 285.776283] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 285.806087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 289.969414] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 290.137842] LNet: 9242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.144484] LNetError: 9242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.164785] LNet: Removed LNI 192.168.201.42@tcp [ 290.857729] Key type .llcrypt unregistered [ 290.862589] Key type ._llcrypt unregistered [ 297.555262] Key type ._llcrypt registered [ 297.559179] Key type .llcrypt registered [ 299.093943] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:52:01 (1769752321) [ 299.517625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 299.535855] alg: No test for adler32 (adler32-zlib) [ 300.484704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.534378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.547860] LNet: 9649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.571674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.616909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.631369] LNet: 9656:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 306.820543] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:52:09 (1769752329) [ 307.072298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.090728] LNet: 10233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.111908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.174904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 313.651722] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:52:15 (1769752335) [ 313.849897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.890762] LNet: 10817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.942370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.096868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 321.197832] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:52:23 (1769752343) [ 321.433287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.450120] LNet: 11403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.465969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.529430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 326.998819] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:52:29 (1769752349) [ 327.078889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.096266] LNet: 11979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.116828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.206256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 327.570071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 334.271973] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:52:36 (1769752356) [ 334.392902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.432929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.545908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 334.997364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 335.358832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 341.557659] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:52:43 (1769752363) [ 341.751483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.809560] LNet: 13176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.814861] LNet: 13176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 341.838495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.935872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 347.474914] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:52:50 (1769752370) [ 347.626761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.672753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.748566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 347.785303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 348.319838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 348.353219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 348.774353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 348.818400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 349.116298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 349.154966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 349.540019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 349.586754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 350.061679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 350.118733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 350.611865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 350.662683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 351.144118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 351.197838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 351.637676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 351.679466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 352.155275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 352.211391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 352.776976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 352.823110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 353.172879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 353.205804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.673556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 353.737735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 359.705850] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:53:02 (1769752382) [ 359.894734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.929493] LNet: 14444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.937322] LNet: 14444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 359.957152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.074066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 360.125269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 367.101391] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:53:09 (1769752389) [ 367.321316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.393872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.508200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 367.549625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 373.454386] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:53:15 (1769752395) [ 373.652269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.694521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.800495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 373.861743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 379.867545] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:53:22 (1769752402) [ 379.989156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.045777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.153512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 380.189759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 387.048415] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:53:29 (1769752409) [ 387.191263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.254529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.306829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 387.343723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 394.454658] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:53:36 (1769752416) [ 394.768337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.794927] LNet: 17376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.801722] LNet: 17376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 394.821056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.918438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 394.956978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 401.453685] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:53:43 (1769752423) [ 401.730877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.788092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.920977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 402.340292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.404231] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:53:51 (1769752431) [ 409.512614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.596843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.797504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.006882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.543804] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:53:58 (1769752438) [ 416.806149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.851685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.967807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.206849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 422.574292] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:54:05 (1769752445) [ 422.839712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.888285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.231917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.374877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 430.231861] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:54:12 (1769752452) [ 430.392981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.450719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.562991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.596317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 439.884611] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:54:21 (1769752461) [ 440.242257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.307153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.422117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.513819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.966948] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:54:31 (1769752471) [ 449.067511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.102331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.154973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 456.599307] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:54:38 (1769752478) [ 456.782514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.864260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.973760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 464.155282] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:54:46 (1769752486) [ 464.359958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.393554] LNet: 22814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.398801] LNet: 22814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 464.418185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.491226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 464.919944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 471.768153] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:54:54 (1769752494) [ 472.116885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.199166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.305397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 472.333884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 478.893731] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:55:00 (1769752500) [ 479.090571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.162610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.274891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 479.314573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 484.850479] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:55:07 (1769752507) [ 485.076884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.108550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.180492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 485.204116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 491.724431] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:55:14 (1769752514) [ 491.827085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.866748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.918119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 491.940894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 497.824541] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:55:20 (1769752520) [ 498.061359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.111517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.155546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 498.206810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 498.220976] LNetError: 25764:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 498.256339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 498.290938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 503.580897] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:55:25 (1769752525) [ 503.687397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.731936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.815603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jL6MYz9pPM [ 503.882552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jL6MYz9pPM [ 503.929939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jL6MYz9pPM [ 509.802529] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:55:32 (1769752532) [ 510.009403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.076814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.150161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 510.194717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 510.252088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 515.616671] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:55:38 (1769752538) [ 515.726550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.778669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.840722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 518.521271] LNet: Message forwarding will be enabled in 2s [ 521.635639] LNet: Message forwarding enabled [ 522.085406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 522.108968] LNet: Message forwarding will stop in 2s to 6s [ 526.673109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 527.223435] LNet: Message forwarding disabled [ 529.413651] LNet: Message forwarding will be enabled in 2s [ 533.407065] LNet: Message forwarding enabled [ 535.887448] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:55:58 (1769752558) [ 536.163576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.586964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.676956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 545.026699] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:56:07 (1769752567) [ 545.112986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.188862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.305965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 545.366355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 545.431552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 545.484900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 545.567877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 545.606481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 545.644963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 545.684843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 545.716323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 545.759965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 545.788354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 545.816978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 545.848941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 545.884509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 545.921850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 545.963609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 545.995691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 546.043991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 546.096938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 546.116899] LNetError: 28833:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 552.271473] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:56:14 (1769752574) [ 552.349410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.398613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.525142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.753598] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:56:21 (1769752581) [ 560.228249] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg142-server [ 561.998568] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:56:24 (1769752584) [ 562.128141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.209748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.299784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 562.339279] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 564.528944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[43-47]@tcp [ 564.566603] LNet: 30549:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 564.658093] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.43@tcp is being used as a gateway but routing feature is not turned on [ 564.673632] LNetError: 30336:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.43@tcp has gone from up to down [ 565.613306] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 565.621699] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 566.654520] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.43@tcp is being used as a gateway but routing feature is not turned on [ 566.686689] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 568.741237] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 568.755582] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 572.150018] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:56:34 (1769752594) [ 572.851645] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 572.872255] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 574.194712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg142-server [ 576.029571] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:56:38 (1769752598) [ 576.217605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.276565] LNetError: 31476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.302813] LNet: Removed LNI 192.168.201.42@tcp [ 576.359154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.529254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 576.595368] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 577.749327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[43-48/2]@tcp [ 577.782424] LNet: 31601:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 577.793141] LNet: 31601:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 577.835188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[43-48/2]@tcp [ 577.841056] LNetError: 31487:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.43@tcp has gone from up to down [ 577.876840] LNetError: 31487:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 584.974756] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:56:47 (1769752607) [ 585.366847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.402863] LNetError: 32193:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.418478] LNet: Removed LNI 192.168.201.42@tcp [ 585.434594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.518754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 585.586658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.615906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.670947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.736161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.768810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.813922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 585.903979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.943819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.099563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 586.186962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.231225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.304128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 586.373599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.402733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.491832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 586.592910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.671655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.723185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 586.805391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.858613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.942525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.029519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.071264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.159907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.290694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.336985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.409564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.511948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.560964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.622901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 587.689489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.737539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.814406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 593.991872] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:56:56 (1769752616) [ 595.876557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg142-server [ 597.594334] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:56:59 (1769752619) [ 599.427849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg142-server [ 601.198676] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:57:03 (1769752623) [ 601.372091] LNet: 33593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.379208] LNet: 33593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 25 previous similar messages [ 601.873742] Key type .llcrypt unregistered [ 601.877728] Key type ._llcrypt unregistered [ 608.178079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 608.331556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 608.336455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 608.339971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.958591] Key type ._llcrypt registered [ 610.963457] Key type .llcrypt registered [ 611.119184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.139495] alg: No test for adler32 (adler32-zlib) [ 612.093764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.195982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 612.244841] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 612.906511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 612.928545] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.839892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 614.856402] LNet: Removed LNI 10.1.2.3@tcp [ 614.977471] LNet: 34311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.983554] LNetError: 34311:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.997742] LNet: Removed LNI 192.168.201.42@tcp [ 615.744765] Key type .llcrypt unregistered [ 615.747513] Key type ._llcrypt unregistered [ 625.898754] Key type ._llcrypt registered [ 625.900578] Key type .llcrypt registered [ 627.547266] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:57:29 (1769752649) [ 628.246848] Key type .llcrypt unregistered [ 628.249442] Key type ._llcrypt unregistered [ 635.401840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 635.686974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 635.690432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 635.697651] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 638.497310] Key type ._llcrypt registered [ 638.500624] Key type .llcrypt registered [ 638.681386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 638.689934] alg: No test for adler32 (adler32-zlib) [ 639.781788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.834848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.856164] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 641.086105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.115881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 644.940152] LNet: 35792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.946865] LNetError: 35792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.964102] LNet: Removed LNI 192.168.201.42@tcp [ 644.981191] LNet: Removed LNI 10.1.2.3@tcp [ 645.780690] Key type .llcrypt unregistered [ 645.782644] Key type ._llcrypt unregistered [ 659.280536] Key type ._llcrypt registered [ 659.284907] Key type .llcrypt registered [ 660.517601] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:58:02 (1769752682) [ 661.283713] Key type .llcrypt unregistered [ 661.286474] Key type ._llcrypt unregistered [ 668.011471] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.114069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.122679] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.130472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.728547] Key type ._llcrypt registered [ 670.733182] Key type .llcrypt registered [ 670.928265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.944135] alg: No test for adler32 (adler32-zlib) [ 671.975407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.069737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 672.121813] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 673.740482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.757658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 675.913604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 675.937458] LNet: Removed LNI 10.1.2.3@tcp [ 675.947994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 675.973662] LNet: Removed LNI 10.1.2.31@tcp [ 675.990971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.012107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 676.742718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 676.761648] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 677.665639] LNet: 37705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.670424] LNetError: 37705:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 677.674413] LNet: Removed LNI 10.1.2.3@tcp [ 677.686422] LNet: Removed LNI 10.1.2.31@tcp [ 678.208636] Key type .llcrypt unregistered [ 678.210296] Key type ._llcrypt unregistered [ 691.913672] Key type ._llcrypt registered [ 691.915388] Key type .llcrypt registered [ 693.185836] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:58:35 (1769752715) [ 693.834335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 693.857915] alg: No test for adler32 (adler32-zlib) [ 694.814229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.874709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.895143] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 695.750980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.781716] LNet: 38655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.785597] LNetError: 38655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 695.808319] LNet: Removed LNI 192.168.201.42@tcp [ 695.856422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.936118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.42@tcp [ 695.975236] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 697.003147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.42@tcp [ 697.040788] LNet: Removed LNI 192.168.201.42@tcp [ 697.124427] LNet: 38790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.672738] Key type .llcrypt unregistered [ 697.678836] Key type ._llcrypt unregistered [ 710.324870] Key type ._llcrypt registered [ 710.330124] Key type .llcrypt registered [ 711.722849] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:58:53 (1769752733) [ 714.159069] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg142-server [ 716.062132] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:58:58 (1769752738) [ 716.544269] Key type .llcrypt unregistered [ 716.548040] Key type ._llcrypt unregistered [ 724.009784] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.288075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.293258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.298833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.824769] Key type ._llcrypt registered [ 726.832632] Key type .llcrypt registered [ 727.006262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.018480] alg: No test for adler32 (adler32-zlib) [ 727.893210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.943489] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 728.618557] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 730.778147] LNet: 40619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.784257] LNetError: 40619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.788470] LNet: Removed LNI 192.168.201.42@tcp1 [ 730.802468] LNet: Removed LNI 10.1.2.3@tcp2 [ 731.380201] Key type .llcrypt unregistered [ 731.384140] Key type ._llcrypt unregistered [ 743.231787] Key type ._llcrypt registered [ 743.235414] Key type .llcrypt registered [ 744.602158] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:59:26 (1769752766) [ 744.946142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 744.959685] alg: No test for adler32 (adler32-zlib) [ 745.956543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.018744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.056749] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 746.800338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.819687] LNet: 41584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.824567] LNetError: 41584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.839214] LNet: Removed LNI 192.168.201.42@tcp [ 746.861141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.950374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 747.005984] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 753.474783] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:59:35 (1769752775) [ 753.637635] LNet: 42172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.646200] LNetError: 42172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.686507] LNet: Removed LNI 192.168.201.42@tcp [ 754.247983] Key type .llcrypt unregistered [ 754.254950] Key type ._llcrypt unregistered [ 760.349101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.501495] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.516279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.527511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 763.168586] Key type ._llcrypt registered [ 763.170754] Key type .llcrypt registered [ 763.350618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.370594] alg: No test for adler32 (adler32-zlib) [ 764.363675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.422470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.442558] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 765.258476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 765.276543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.939939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.990328] LNet: 42915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.001220] LNetError: 42915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.012524] LNet: Removed LNI 192.168.201.42@tcp [ 768.035280] LNet: Removed LNI 10.1.2.3@tcp [ 768.087790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.221556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 768.285587] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 768.289522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.971632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.994357] LNet: 42942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.000334] LNetError: 42942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.005990] LNetError: 42942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 769.013444] LNet: Removed LNI 192.168.201.42@tcp [ 769.032479] LNet: Removed LNI 10.1.2.3@tcp [ 769.066704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.147652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 769.186617] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 769.194796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 769.884418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 769.904103] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 771.217703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.241485] LNet: 43105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.246176] LNetError: 43105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.252419] LNetError: 43105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 771.260118] LNet: Removed LNI 192.168.201.42@tcp [ 771.271198] LNet: Removed LNI 10.1.2.3@tcp [ 771.299079] LNet: Removed LNI 192.168.201.42@tcp2 [ 771.345590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.482377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 771.537945] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 771.552821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.566911] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 778.640411] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:00:01 (1769752801) [ 778.737352] LNet: 43693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.746610] LNetError: 43693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.753289] LNetError: 43693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 778.760332] LNet: Removed LNI 192.168.201.42@tcp [ 778.762409] LNet: Removed LNI 10.1.2.3@tcp [ 778.770522] LNet: Removed LNI 192.168.201.42@tcp2 [ 779.375963] Key type .llcrypt unregistered [ 779.377616] Key type ._llcrypt unregistered [ 785.367666] Key type ._llcrypt registered [ 785.371513] Key type .llcrypt registered [ 785.622661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.641787] alg: No test for adler32 (adler32-zlib) [ 786.661831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.752221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 786.792303] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 787.704649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.722901] LNet: 44092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.727747] LNetError: 44092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.744132] LNet: Removed LNI 192.168.201.42@tcp [ 787.771554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.827338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 787.845853] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 788.406310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.447275] LNet: 44122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.460407] LNetError: 44122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.485963] LNet: Removed LNI 192.168.201.42@tcp [ 788.510644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.602949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 788.652229] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 789.323420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.383275] LNet: Removed LNI 192.168.201.42@tcp [ 789.407226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.503491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 789.533139] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 790.028549] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 790.131872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 790.136485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 790.193091] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 792.697437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.717387] LNet: 44231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.724572] LNet: 44231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 792.730734] LNetError: 44231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.735989] LNetError: 44231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.763711] LNet: Removed LNI 192.168.201.42@tcp [ 792.820531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.900482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 792.953203] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 799.486441] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:00:21 (1769752821) [ 799.610081] LNet: 44821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.613902] LNetError: 44821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.635720] LNet: Removed LNI 192.168.201.42@tcp [ 800.200981] Key type .llcrypt unregistered [ 800.204793] Key type ._llcrypt unregistered [ 807.447877] Key type ._llcrypt registered [ 807.449376] Key type .llcrypt registered [ 807.638198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.655673] alg: No test for adler32 (adler32-zlib) [ 808.639620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.713427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 808.733143] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 809.734392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.752675] LNet: 45222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.756925] LNetError: 45222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.773918] LNet: Removed LNI 192.168.201.42@tcp [ 809.793708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.890147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 809.933547] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 816.301919] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:00:38 (1769752838) [ 816.397886] LNet: 45806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.407038] LNetError: 45806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.432521] LNet: Removed LNI 192.168.201.42@tcp [ 817.133401] Key type .llcrypt unregistered [ 817.136350] Key type ._llcrypt unregistered [ 824.366473] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.500757] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.508922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.523344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 827.032068] Key type ._llcrypt registered [ 827.038457] Key type .llcrypt registered [ 827.204579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.228804] alg: No test for adler32 (adler32-zlib) [ 828.148831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.198771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.224122] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 829.124333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.143678] LNet: 46260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.150213] LNetError: 46260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.165869] LNet: Removed LNI 192.168.201.42@tcp [ 829.184962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.281924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 829.313352] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 836.625130] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:00:58 (1769752858) [ 836.711490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.766775] LNet: 46850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.776411] LNetError: 46850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.791878] LNet: Removed LNI 192.168.201.42@tcp [ 836.823251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.933695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 844.185841] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:01:06 (1769752866) [ 844.272995] LNet: 47444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.966556] Key type .llcrypt unregistered [ 844.968476] Key type ._llcrypt unregistered [ 851.137800] Key type ._llcrypt registered [ 851.154540] Key type .llcrypt registered [ 851.425879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.443615] alg: No test for adler32 (adler32-zlib) [ 852.396881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.475432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 852.494242] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 853.521105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.544729] LNet: 47846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.551767] LNetError: 47846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.572658] LNet: Removed LNI 192.168.201.42@tcp [ 853.601205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.638512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 853.654949] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 860.168580] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:01:22 (1769752882) [ 860.300200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.332527] LNet: 48435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.335946] LNetError: 48435:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.360729] LNet: Removed LNI 192.168.201.42@tcp [ 860.389303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.467881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 866.872238] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:01:29 (1769752889) [ 868.320426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 870.223439] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:01:32 (1769752892) [ 871.796091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 873.535727] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:01:35 (1769752895) [ 875.234305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 877.040082] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:01:39 (1769752899) [ 878.723059] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 880.472385] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:01:42 (1769752902) [ 882.409165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 884.102705] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:01:46 (1769752906) [ 884.428209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.451352] LNet: 50491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.474368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.551373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.588271] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 885.311988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.42 [ 885.326604] LNet: 50625:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 885.333372] LNet: Removed LNI 192.168.201.42@tcp [ 891.116401] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:01:53 (1769752913) [ 891.311229] LNet: 51196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.323542] LNetError: 51196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.346248] LNet: Removed LNI 192.168.201.42@tcp [ 892.091661] Key type .llcrypt unregistered [ 892.093676] Key type ._llcrypt unregistered [ 899.017078] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.285762] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.288344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.292880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.085643] Key type ._llcrypt registered [ 902.087939] Key type .llcrypt registered [ 902.364323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.398606] alg: No test for adler32 (adler32-zlib) [ 903.881453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.299129] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:02:12 (1769752932) [ 910.384400] LNet: 52091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.389696] LNetError: 52091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.403991] LNet: Removed LNI 10.1.2.3@tcp [ 911.104727] Key type .llcrypt unregistered [ 911.106274] Key type ._llcrypt unregistered [ 917.900202] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.197591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.219769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.233918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 921.094717] Key type ._llcrypt registered [ 921.098340] Key type .llcrypt registered [ 921.401509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.425279] alg: No test for adler32 (adler32-zlib) [ 922.415494] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 929.024575] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:02:31 (1769752951) [ 929.111285] LNet: 52979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.113927] LNetError: 52979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.125165] LNet: Removed LNI 10.1.2.3@tcp [ 929.813576] Key type .llcrypt unregistered [ 929.818493] Key type ._llcrypt unregistered [ 936.430894] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 936.686237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 936.698667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 936.720731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 938.354087] Key type ._llcrypt registered [ 938.356807] Key type .llcrypt registered [ 938.510915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.521551] alg: No test for adler32 (adler32-zlib) [ 939.516726] LNet: 53276:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 939.533339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.206357] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:02:48 (1769752968) [ 946.523468] LNet: 53868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.545034] LNetError: 53868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.580305] LNet: Removed LNI 10.1.2.3@tcp [ 947.228481] Key type .llcrypt unregistered [ 947.230970] Key type ._llcrypt unregistered [ 953.181798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.360368] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.370388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.379972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.065190] Key type ._llcrypt registered [ 956.066730] Key type .llcrypt registered [ 956.220693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.232359] alg: No test for adler32 (adler32-zlib) [ 957.177815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.239874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 966.832808] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:03:08 (1769752988) [ 967.328471] LNet: 55093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.334749] LNetError: 55093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 967.349551] LNet: Removed LNI 10.1.2.3@tcp [ 968.194755] Key type .llcrypt unregistered [ 968.197304] Key type ._llcrypt unregistered [ 975.263700] Key type ._llcrypt registered [ 975.265713] Key type .llcrypt registered [ 975.523634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.552092] alg: No test for adler32 (adler32-zlib) [ 976.546975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 976.583923] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 985.219669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 985.896725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 985.926306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 985.957651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 985.972769] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 986.195857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.236110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 986.278701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.313518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.348799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.409170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.455972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.511196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 986.534410] LNet: 55527:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.571666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 986.633322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 986.697940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 986.747258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 986.802677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 986.832798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 986.848358] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 986.863185] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 986.884948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 987.472203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 987.566282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 987.650392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 988.070763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.098217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 988.123375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.152540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.177372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.202627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.232338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.255702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.271810] LNet: 55636:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.288987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.315822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.346805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.369809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.399497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.456759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.502886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 989.207426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 989.259754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 989.311806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 989.330279] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 989.337974] LNet: Skipped 1 previous similar message [ 989.564052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.591564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 989.619848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.646667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.684446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.709503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.734437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.765578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.779271] LNet: 55745:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.793392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.825113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.887272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.926228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.962563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.992319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.016944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 990.597234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 990.634544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 990.688648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 990.958425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.984610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 991.010333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.051712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.084874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.111909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.145363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.198813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 991.251788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 991.292992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 991.323448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 991.364841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.405517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.440908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.481255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 992.195749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 992.227717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 992.272438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 992.306932] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 992.317546] LNet: Skipped 1 previous similar message [ 992.687387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.719610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 992.773935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.822687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.864343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.907618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.946897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.984851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.026819] LNet: 55964:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 993.038593] LNet: 55964:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 993.062959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.125510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.184565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.226243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.255632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.321954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.408625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 994.055923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 994.116395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 994.194787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 994.503803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.552519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 994.584976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.623877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.659550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.693628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.730226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.772771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.818967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.865861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.897738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.922269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.947548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.974624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.018732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 995.237519] LNet: 56113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.242803] LNetError: 56113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.258811] LNet: Removed LNI 192.168.201.42@tcp [ 995.568072] Key type .llcrypt unregistered [ 995.569560] Key type ._llcrypt unregistered [ 1014.053028] Key type ._llcrypt registered [ 1014.056746] Key type .llcrypt registered [ 1015.469534] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:03:57 (1769753037) [ 1016.574354] Key type .llcrypt unregistered [ 1016.578128] Key type ._llcrypt unregistered [ 1024.031098] Key type ._llcrypt registered [ 1024.035172] Key type .llcrypt registered [ 1024.235641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.252406] alg: No test for adler32 (adler32-zlib) [ 1025.249895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1025.282680] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1032.644517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1044.217413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.303746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1044.352968] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1046.250391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 1046.281398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 1 [ 1046.317025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 1046.356578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 1 [ 1046.393839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_interrupt -r 1 [ 1046.424310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 1 [ 1046.458435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_interrupt -r 1 [ 1046.492824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 1 [ 1046.536092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1046.569487] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1046.585105] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1046.879876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.922855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.948824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.977482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.006131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.042876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.083918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.120436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.139914] LNet: 57573:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.160493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.200970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.241961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.278634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.317895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.372592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.422618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.965282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 1048.027773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 1 [ 1048.087996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 1048.134412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 1 [ 1048.197817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_dropped -r 1 [ 1048.237307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 1 [ 1048.310627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_dropped -r 1 [ 1048.383899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 1 [ 1048.482780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1048.517373] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1048.530667] LNet: Skipped 2 previous similar messages [ 1048.891553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.944169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.992762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1049.061424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1049.114743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1049.153936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1049.228946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1049.289298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1049.323960] LNet: 57695:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1049.347532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1049.414764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1049.467501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.502290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.538764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.578724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.579802] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1049.601297] LNet: Skipped 3 previous similar messages [ 1049.628133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1050.175911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 1050.212409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 1 [ 1050.276318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 1050.315761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 1 [ 1050.373646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_aborted -r 1 [ 1050.418549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 1 [ 1050.459344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_aborted -r 1 [ 1050.504686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 1 [ 1050.540557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1050.906593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1050.957698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1050.996661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.035597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.075909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.107972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.145539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.191338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.218683] LNet: 57816:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.254882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.292689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.369742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.441590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.512955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.571467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.644417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.405767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 1052.478912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 1 [ 1052.535955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 1052.603185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 1 [ 1052.692562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_no_route -r 1 [ 1052.749673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 1 [ 1052.815750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_no_route -r 1 [ 1052.861496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 1 [ 1052.919076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1052.946567] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1052.952121] LNet: Skipped 3 previous similar messages [ 1053.220876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.277980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1053.334461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.369232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.412583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.456939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.516513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.569915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.621800] LNet: 57938:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.663254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.707907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.731926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.770927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.806916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.841431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.876268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1054.789393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 1054.839054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 1 [ 1054.875640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 1054.933472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 1 [ 1054.990897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_timeout -r 1 [ 1055.036826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 1 [ 1055.067865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_timeout -r 1 [ 1055.098945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 1 [ 1055.124982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1055.537634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.600397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1055.639761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.686788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.741533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.773647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.800745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.829896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.851855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.872759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.898837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.929348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.952339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.978399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.010861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.606535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 1056.656729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_error -r 1 [ 1056.708382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 1056.754674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_error -r 1 [ 1056.838801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_error -r 1 [ 1056.934528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_error -r 1 [ 1057.015927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_error -r 1 [ 1057.095248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_error -r 1 [ 1057.158202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1057.234679] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1057.244691] LNet: Skipped 5 previous similar messages [ 1057.774724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.861866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1057.962910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.022148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.085316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.151110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.216187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.286987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.347581] LNet: 58180:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.357367] LNet: 58180:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1058.399633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.440169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.499603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.550023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.614756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.670160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.714794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1064.936073] LNet: 58255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.954707] LNetError: 58255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.969098] LNet: Removed LNI 192.168.201.42@tcp [ 1066.021798] LNet: Removed LNI 192.168.201.42@tcp1 [ 1066.589567] Key type .llcrypt unregistered [ 1066.591226] Key type ._llcrypt unregistered [ 1086.776592] Key type ._llcrypt registered [ 1086.778297] Key type .llcrypt registered [ 1088.187399] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:05:10 (1769753110) [ 1089.316360] Key type .llcrypt unregistered [ 1089.320519] Key type ._llcrypt unregistered [ 1097.381665] Key type ._llcrypt registered [ 1097.387188] Key type .llcrypt registered [ 1097.580709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.603523] alg: No test for adler32 (adler32-zlib) [ 1098.549932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1098.570934] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1106.998164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1107.704319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1107.735727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1107.770942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1107.794659] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1108.088969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.126216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1108.171984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.220603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.271882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.341043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.390464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.443962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.480765] LNet: 59500:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.520775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.574042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.635545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.695881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.740812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.789822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.828472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.524661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1109.571142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1109.609727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1109.640557] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1109.945672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.994092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.087753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.147874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.208278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.246943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.294551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.336380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.355809] LNet: 59609:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.394467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.435990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.482650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.552727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.607327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.643600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.682122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.312491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1111.412821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1111.462711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1111.497660] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1111.793133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.828927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.876596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.902626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.935211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.965311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.994808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.022747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.037736] LNet: 59719:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.052877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.085776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.112691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.147819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.178161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.217521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.262641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.611551] LNet: 59759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.625773] LNetError: 59759:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.652291] LNet: Removed LNI 192.168.201.42@tcp [ 1113.127611] Key type .llcrypt unregistered [ 1113.131899] Key type ._llcrypt unregistered [ 1128.690962] Key type ._llcrypt registered [ 1128.695772] Key type .llcrypt registered [ 1130.252702] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:05:52 (1769753152) [ 1131.185422] Key type .llcrypt unregistered [ 1131.187326] Key type ._llcrypt unregistered [ 1138.701141] Key type ._llcrypt registered [ 1138.706352] Key type .llcrypt registered [ 1139.022937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.055375] alg: No test for adler32 (adler32-zlib) [ 1140.138300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1140.172308] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1148.868580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1159.825975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.888665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1159.903072] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1161.159734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1161.203273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 1 [ 1161.241577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1161.273531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 1 [ 1161.311598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_dropped -r 1 [ 1161.356228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 1 [ 1161.411583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_dropped -r 1 [ 1161.455607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 1 [ 1161.506354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1161.527967] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1161.538763] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1161.811489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.837467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1161.862927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.897578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.925788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.959632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.993717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.031584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.069957] LNet: 61220:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.097408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.140873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.171517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.199151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.240875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.292552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.332979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.383866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.138477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1163.231301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_error -r 1 [ 1163.286030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1163.346467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_error -r 1 [ 1163.415429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_error -r 1 [ 1163.506985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_error -r 1 [ 1163.576644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_error -r 1 [ 1163.638990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_error -r 1 [ 1163.712527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1163.752564] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1163.771980] LNet: Skipped 2 previous similar messages [ 1164.210602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.265429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1164.325110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.383050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.455937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.491341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.551742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1164.621368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1164.648566] LNet: 61345:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1164.671365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1164.698790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.723207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.766647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.805443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.845364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.900465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.952699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.593362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1165.634502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_timeout -r 1 [ 1165.682147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1165.718258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_timeout -r 1 [ 1165.756486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_timeout -r 1 [ 1165.802136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_timeout -r 1 [ 1165.844820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_timeout -r 1 [ 1165.886800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_timeout -r 1 [ 1165.926388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.142@tcp [ 1165.944466] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1165.950405] LNet: Skipped 1 previous similar message [ 1166.167611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.211762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1166.266332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.314893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.359238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.386293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.413512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.443958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.469928] LNet: 61469:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.493785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.534846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.581781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.611967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.651861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.701561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.741852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.783764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.483602] LNet: 61547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.495983] LNetError: 61547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.509776] LNet: Removed LNI 192.168.201.42@tcp [ 1172.522578] LNet: Removed LNI 192.168.201.42@tcp1 [ 1173.087895] Key type .llcrypt unregistered [ 1173.096833] Key type ._llcrypt unregistered [ 1191.750790] Key type ._llcrypt registered [ 1191.755967] Key type .llcrypt registered [ 1193.339410] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:06:55 (1769753215) [ 1193.904439] Key type .llcrypt unregistered [ 1193.907154] Key type ._llcrypt unregistered [ 1201.443289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1201.684035] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1201.692277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1201.704675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1203.482747] Key type ._llcrypt registered [ 1203.484911] Key type .llcrypt registered [ 1203.674486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1203.686360] alg: No test for adler32 (adler32-zlib) [ 1204.664715] LNet: 62616:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1204.683875] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1204.754221] LNet: 62630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.764431] LNetError: 62630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.786964] LNet: Removed LNI 192.168.201.42@tcp [ 1205.486552] Key type .llcrypt unregistered [ 1205.488069] Key type ._llcrypt unregistered [ 1212.868142] Key type ._llcrypt registered [ 1212.869578] Key type .llcrypt registered [ 1213.085316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.106835] alg: No test for adler32 (adler32-zlib) [ 1214.067161] LNet: 62886:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1214.078944] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1214.083580] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1214.125972] LNet: 62900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.133837] LNetError: 62900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.139910] LNet: Removed LNI 192.168.201.42@tcp [ 1214.150855] LNet: Removed LNI 10.1.2.3@tcp1 [ 1214.853596] Key type .llcrypt unregistered [ 1214.856567] Key type ._llcrypt unregistered [ 1222.189466] Key type ._llcrypt registered [ 1222.192210] Key type .llcrypt registered [ 1222.467863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.498691] alg: No test for adler32 (adler32-zlib) [ 1223.481784] LNet: 63155:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1223.501935] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1223.505121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1223.566244] LNet: 63169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.579583] LNetError: 63169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.585845] LNet: Removed LNI 192.168.201.42@tcp [ 1223.610634] LNet: Removed LNI 10.1.2.3@tcp [ 1224.530373] Key type .llcrypt unregistered [ 1224.535893] Key type ._llcrypt unregistered [ 1230.857627] Key type ._llcrypt registered [ 1230.861899] Key type .llcrypt registered [ 1231.134372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1231.185522] alg: No test for adler32 (adler32-zlib) [ 1232.200444] LNet: 63442:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1232.251321] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1232.254409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1232.316590] LNet: 63456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.328516] LNetError: 63456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.331592] LNet: Removed LNI 192.168.201.42@tcp [ 1232.362038] LNet: Removed LNI 10.1.2.3@tcp [ 1232.986679] Key type .llcrypt unregistered [ 1232.989182] Key type ._llcrypt unregistered [ 1240.033458] Key type ._llcrypt registered [ 1240.057075] Key type .llcrypt registered [ 1240.418410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1240.440468] alg: No test for adler32 (adler32-zlib) [ 1241.404317] LNet: 63716:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1241.428251] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1241.431995] LNetError: 63716:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1241.438680] LNetError: Error -100 starting up LNI tcp [ 1241.449884] LNet: Removed LNI 192.168.201.42@tcp [ 1242.318559] Key type .llcrypt unregistered [ 1242.322122] Key type ._llcrypt unregistered [ 1254.839235] Key type ._llcrypt registered [ 1254.840968] Key type .llcrypt registered [ 1256.141610] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:07:58 (1769753278) [ 1257.168555] Key type .llcrypt unregistered [ 1257.173080] Key type ._llcrypt unregistered [ 1264.049734] Key type ._llcrypt registered [ 1264.055833] Key type .llcrypt registered [ 1264.428709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.486566] alg: No test for adler32 (adler32-zlib) [ 1265.442804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.509662] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1274.809127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1275.785885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1275.846253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1275.917714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1275.955679] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1276.170430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.200393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1276.242907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.283236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.331915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.370693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.405721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.448746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.486944] LNet: 64965:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.520860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.564583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.624925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.665484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.711986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.737376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.012935] LNet: 65003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.024596] LNetError: 65003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.039229] LNet: Removed LNI 192.168.201.42@tcp [ 1277.546413] Key type .llcrypt unregistered [ 1277.548785] Key type ._llcrypt unregistered [ 1299.033599] Key type ._llcrypt registered [ 1299.036288] Key type .llcrypt registered [ 1299.306424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1299.335719] alg: No test for adler32 (adler32-zlib) [ 1300.373541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1300.407814] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1308.859164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1318.528197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1318.582198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1318.620930] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1320.157325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1320.191229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e network_timeout -r 1 [ 1320.232398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1320.268202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e network_timeout -r 1 [ 1320.298232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e network_timeout -r 1 [ 1320.335514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e network_timeout -r 1 [ 1320.375299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e network_timeout -r 1 [ 1320.425536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e network_timeout -r 1 [ 1320.476367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1320.511167] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1320.778835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.813940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1320.842649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.880306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.907580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.928817] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1320.931879] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1320.938467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.971320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1320.999870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.017232] LNet: 65877:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.035886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.085650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.118199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.138681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.160976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.189975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.408879] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1326.813819] LNet: 65950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.819578] LNetError: 65950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.839637] LNet: Removed LNI 192.168.201.42@tcp [ 1327.849848] LNetError: 65950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.880665] LNet: Removed LNI 192.168.201.42@tcp1 [ 1328.424473] Key type .llcrypt unregistered [ 1328.428647] Key type ._llcrypt unregistered [ 1347.150151] Key type ._llcrypt registered [ 1347.152597] Key type .llcrypt registered [ 1348.576965] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:09:30 (1769753370) [ 1349.117220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.147124] alg: No test for adler32 (adler32-zlib) [ 1350.103202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.196982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.245102] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1351.057266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.090301] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1351.973737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1352.036885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1352.071146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1352.092741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1352.113403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1352.139820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1352.167870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1352.185559] LNetError: 67046:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1352.230425] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1352.237372] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1354.275166] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1354.601481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1358.367544] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1359.348144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1359.402763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1359.465391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.485908] LNet: 67153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.495634] LNetError: 67153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.545892] LNet: Removed LNI 192.168.201.42@tcp [ 1360.555726] LNetError: 67153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.584762] LNet: Removed LNI 192.168.201.42@tcp1 [ 1360.649526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.733560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.782600] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1361.680156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1361.698415] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1362.758364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1362.798467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.888453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1362.952605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1362.992339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1363.024757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1363.068353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1363.105713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1363.122947] LNetError: 67426:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1363.743540] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1366.801679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1369.887597] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1369.894716] LNet: Skipped 1 previous similar message [ 1374.832001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1374.865570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1374.896980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.631782] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:10:04 (1769753404) [ 1382.786717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.815943] LNet: 68140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.823451] LNetError: 68140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.838714] LNet: Removed LNI 192.168.201.42@tcp [ 1382.861557] LNet: Removed LNI 192.168.201.42@tcp1 [ 1382.886116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.965479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.024449] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1384.010697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.051487] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1385.189340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1385.269751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1385.319576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.366340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.410270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1385.449295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.491794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 0 [ 1385.612386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1386.016402] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.42@tcp [ 1386.019093] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1386.021903] LNet: Skipped 1 previous similar message [ 1401.381986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1401.607968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1401.638051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.684503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 500 [ 1401.793288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1408.441942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1408.501809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.524768] LNet: 68635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.531045] LNetError: 68635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.535771] LNetError: 68635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1409.567838] LNet: Removed LNI 192.168.201.42@tcp [ 1409.589700] LNet: Removed LNI 192.168.201.42@tcp1 [ 1409.616340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.689447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.723518] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1410.581554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1410.611591] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1411.899887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1411.957741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1411.999982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1412.020206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1412.051250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1412.089154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.121867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.155703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.42@tcp --health 0 [ 1413.727482] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1413.739456] LNet: Skipped 3 previous similar messages [ 1414.622380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.013302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1425.062870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1425.105557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.987951] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:10:54 (1769753454) [ 1432.406631] LNet: 69692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.411935] LNetError: 69692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.416117] LNetError: 69692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.439152] LNet: Removed LNI 192.168.201.42@tcp [ 1433.450834] LNet: Removed LNI 192.168.201.42@tcp1 [ 1434.347370] Key type .llcrypt unregistered [ 1434.351524] Key type ._llcrypt unregistered [ 1441.213915] Key type ._llcrypt registered [ 1441.215554] Key type .llcrypt registered [ 1441.420040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.439236] alg: No test for adler32 (adler32-zlib) [ 1442.455284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1442.482266] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1451.873970] Lustre: DEBUG MARKER: Initial discovery [ 1451.982713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1454.179599] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1454.260945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1454.329369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1454.390050] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1454.443664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1454.482556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.142@tcp -r 1 -m GET -l 3 [ 1454.520880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.142@tcp -r 1 -m GET -e local_error [ 1454.556831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.142@tcp -r 1 -m PUT -l 6 [ 1454.815801] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1456.048401] Lustre: DEBUG MARKER: Force oleg142-client.virtnet to discover 192.168.201.142@tcp (in background) [ 1456.190870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.142@tcp [ 1457.033594] Lustre: DEBUG MARKER: Force oleg142-server to discover 192.168.201.42@tcp [ 1459.231695] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1459.799631] Lustre: DEBUG MARKER: Wait for 70452 [ 1461.567367] Lustre: DEBUG MARKER: Finished wait on 70452 [ 1461.736641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1461.790638] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1461.799409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1462.043520] LNet: 70900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.052180] LNetError: 70900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.068496] LNet: Removed LNI 192.168.201.42@tcp [ 1462.712382] Key type .llcrypt unregistered [ 1462.714034] Key type ._llcrypt unregistered [ 1483.847538] Key type ._llcrypt registered [ 1483.849546] Key type .llcrypt registered [ 1485.438592] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:11:47 (1769753507) [ 1486.134339] Key type .llcrypt unregistered [ 1486.136802] Key type ._llcrypt unregistered [ 1491.695075] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1491.982305] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1491.991449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1491.998097] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1494.512603] Key type ._llcrypt registered [ 1494.516202] Key type .llcrypt registered [ 1494.677677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1494.693167] alg: No test for adler32 (adler32-zlib) [ 1495.658430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.735251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.762240] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1496.832560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1496.859831] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1506.265759] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:12:08 (1769753528) [ 1506.404519] LNet: 72967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.419464] LNetError: 72967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.430059] LNet: Removed LNI 192.168.201.42@tcp [ 1506.459547] LNet: Removed LNI 10.1.2.3@tcp [ 1507.300776] Key type .llcrypt unregistered [ 1507.304661] Key type ._llcrypt unregistered [ 1514.415762] Key type ._llcrypt registered [ 1514.418730] Key type .llcrypt registered [ 1514.731663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1514.751684] alg: No test for adler32 (adler32-zlib) [ 1515.654779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.729900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1515.762515] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1516.541834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1516.573856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1517.687356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.42@tcp [ 1518.751589] LNetError: Unexpected error -22 connecting to 192.168.201.42@tcp at host 192.168.201.42:988 [ 1523.501718] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:12:25 (1769753545) [ 1523.723461] LNet: 74103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.730216] LNetError: 74103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.740397] LNet: Removed LNI 192.168.201.42@tcp [ 1523.753587] LNet: Removed LNI 10.1.2.3@tcp [ 1524.426329] Key type .llcrypt unregistered [ 1524.431859] Key type ._llcrypt unregistered [ 1530.432658] Key type ._llcrypt registered [ 1530.434596] Key type .llcrypt registered [ 1530.544300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1530.552420] alg: No test for adler32 (adler32-zlib) [ 1531.437741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1531.500641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1531.518858] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1532.295841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1532.315264] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 1533.306458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.42@tcp1 --nid 192.168.201.42@tcp2 [ 1538.302612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp1 192.168.201.42@tcp2 [ 1538.335128] LNetError: 74934:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.42@tcp1 to 192.168.201.42@tcp2 - Route table may be misconfigured [ 1538.362463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp2 192.168.201.42@tcp1 [ 1545.898210] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:12:48 (1769753568) [ 1546.127743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.163627] LNet: 75507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.169334] LNetError: 75507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.231446] LNet: Removed LNI 192.168.201.42@tcp1 [ 1547.234800] LNetError: 75507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.252776] LNet: Removed LNI 192.168.201.42@tcp2 [ 1547.299152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.399850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.514122] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1548.177710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1548.193803] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1549.315570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 1549.353426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.42@tcp [ 1549.408955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp1 192.168.201.42@tcp1 [ 1549.546790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.42@tcp -d 192.168.201.42@tcp -e network_timeout [ 1549.635938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -e network_timeout [ 1549.680885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -e network_timeout [ 1549.724357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -e network_timeout [ 1549.755566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1549.777983] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1549.805582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1549.913937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1550.025924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1550.436876] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1556.892309] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:12:58 (1769753578) [ 1557.053103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.123975] LNet: 76366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.134908] LNetError: 76366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.148840] LNet: Removed LNI 192.168.201.42@tcp [ 1558.182553] LNet: Removed LNI 192.168.201.42@tcp1 [ 1558.253530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.368264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1558.710523] LNet: 76405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.102189] Key type .llcrypt unregistered [ 1559.108793] Key type ._llcrypt unregistered [ 1570.673688] Key type ._llcrypt registered [ 1570.676436] Key type .llcrypt registered [ 1572.158790] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:13:14 (1769753594) [ 1572.537058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.551388] alg: No test for adler32 (adler32-zlib) [ 1573.571576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.492989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1577.472668] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:13:19 (1769753599) [ 1577.565934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.603019] LNet: 77494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.650488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.715611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.791964] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1578.537545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1578.556791] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1579.366712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1579.440303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp1 [ 1579.495791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp1 [ 1585.199505] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:13:27 (1769753607) [ 1586.937060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg142-server [ 1588.612918] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:13:30 (1769753610) [ 1590.527363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg142-server [ 1592.145773] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:13:34 (1769753614) [ 1594.053705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg142-server [ 1595.603391] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:13:37 (1769753617) [ 1597.537857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg142-server [ 1599.580829] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:13:41 (1769753621) [ 1601.421227] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg142-server [ 1603.455775] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:13:45 (1769753625) [ 1605.494927] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg142-server [ 1607.012776] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:13:49 (1769753629) [ 1609.825282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg142-server [ 1612.081284] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:13:53 (1769753633) [ 1613.511265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg142-server [ 1615.371951] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:13:57 (1769753637) [ 1617.418737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg142-server [ 1619.510963] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:14:01 (1769753641) [ 1619.661250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.706886] LNet: 81374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.723585] LNetError: 81374:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1619.734645] LNet: Removed LNI 192.168.201.42@tcp [ 1620.774495] LNet: Removed LNI 192.168.201.42@tcp1 [ 1620.821947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.926137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1620.963060] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1621.735352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1621.813805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1621.949940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.976395] LNet: 81525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.985582] LNetError: 81525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.989397] LNetError: 81525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1622.001742] LNet: Removed LNI 192.168.201.42@tcp [ 1622.017907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.050449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1622.082109] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1622.781860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1622.962961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.045945] LNet: Removed LNI 192.168.201.42@tcp [ 1624.085196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.181574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.210494] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1625.076868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1625.199430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1625.422665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.451379] LNet: 81818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.457956] LNet: 81818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1625.461770] LNetError: 81818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.467892] LNetError: 81818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1625.482813] LNet: Removed LNI 192.168.201.42@tcp [ 1625.504466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.568823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1625.596216] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1626.254255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1626.385964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.431112] LNet: Removed LNI 192.168.201.42@tcp [ 1627.500596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.595060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.666428] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1628.486278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1628.545872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1628.715888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.733672] LNetError: 82110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.737559] LNetError: 82110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1629.796778] LNet: Removed LNI 192.168.201.42@tcp [ 1629.813401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.872420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1629.894599] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1630.766725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1632.158968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.211255] LNet: 82265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.222081] LNet: 82265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1633.259559] LNet: Removed LNI 192.168.201.42@tcp [ 1633.287313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.404441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.438371] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1634.515391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1634.654228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1634.969467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.996213] LNetError: 82412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.006776] LNetError: 82412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1636.095191] LNet: Removed LNI 192.168.201.42@tcp [ 1636.148838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.224925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1636.241941] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1636.867461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1637.054819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.135462] LNet: Removed LNI 192.168.201.42@tcp [ 1638.158599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.218334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.247522] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1639.367498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1639.476666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1641.641624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.661236] LNet: 82714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.664528] LNet: 82714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1642.735157] LNet: Removed LNI 192.168.201.42@tcp [ 1642.769313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.866885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1642.926048] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1643.866483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1644.192938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.248660] LNetError: 82859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1644.261225] LNetError: 82859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1644.286635] LNet: Removed LNI 192.168.201.42@tcp [ 1644.345358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.458265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.487717] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1645.394070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1645.577409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1645.927347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.016642] LNet: Removed LNI 192.168.201.42@tcp [ 1647.102372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.243118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1647.328517] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1648.952127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1649.548170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.665462] LNet: Removed LNI 192.168.201.42@tcp [ 1650.729617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.870049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.922612] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1652.447734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1652.590135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.693837] LNet: Removed LNI 192.168.201.42@tcp [ 1652.774821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.871336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.939906] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1654.470020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1664.410690] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:14:46 (1769753686) [ 1664.745470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.781620] LNet: 83989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.787459] LNet: 83989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1664.800218] LNetError: 83989:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.805212] LNetError: 83989:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1664.843668] LNet: Removed LNI 192.168.201.42@tcp [ 1664.877551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.947845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1664.977729] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1665.977602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.002560] LNet: Removed LNI 192.168.201.42@tcp231 [ 1666.018978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.080518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1666.096621] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1666.100299] LNet: 84135:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.398675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1666.446838] LNet: Removed LNI 192.168.201.42@tcp231 [ 1666.466992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1666.496148] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1667.547937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.583921] LNet: Removed LNI 192.168.201.42@tcp231 [ 1667.615979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.699785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1667.734155] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/0] [ 1667.738650] LNet: 84288:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1668.232725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1668.259614] LNet: Removed LNI 192.168.201.42@tcp231 [ 1668.285607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1668.317983] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/0] [ 1669.491926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.536648] LNet: Removed LNI 192.168.201.42@tcp231 [ 1669.551973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.609867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1669.631966] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/60] [ 1669.637808] LNet: 84439:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1670.156271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1670.184793] LNet: Removed LNI 192.168.201.42@tcp231 [ 1670.203711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1670.231683] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/60] [ 1671.043731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.061840] LNet: Removed LNI 192.168.201.42@tcp231 [ 1671.084302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.137425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1671.154208] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1671.470034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1671.498937] LNet: Removed LNI 192.168.201.42@tcp231 [ 1671.516708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1671.538854] LNet: Added LNI 192.168.201.42@tcp231 [8/256/0/180] [ 1679.121818] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:15:01 (1769753701) [ 1679.330990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.357769] LNet: Removed LNI 192.168.201.42@tcp231 [ 1679.377725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.448496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1679.471761] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1680.109107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.147730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.177926] LNet: Removed LNI 192.168.201.42@tcp232 [ 1680.205654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1680.243876] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1680.939930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1680.972061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1680.998377] LNet: Removed LNI 192.168.201.42@tcp232 [ 1681.030881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1681.064839] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1681.874648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1681.913695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1681.957260] LNet: Removed LNI 192.168.201.42@tcp232 [ 1681.979476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1682.010526] LNet: Added LNI 192.168.201.42@tcp232 [8/256/0/180] [ 1682.686932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1682.730603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1682.753864] LNet: Removed LNI 192.168.201.42@tcp232 [ 1682.768392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1688.770567] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:15:11 (1769753711) [ 1689.711072] Key type .llcrypt unregistered [ 1689.714064] Key type ._llcrypt unregistered [ 1695.687670] Key type ._llcrypt registered [ 1695.689439] Key type .llcrypt registered [ 1696.102656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1696.116940] alg: No test for adler32 (adler32-zlib) [ 1696.979895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1696.996397] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1704.055236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1716.085464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.133251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1716.167499] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1718.349678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_interrupt -r 2 [ 1718.440716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 2 [ 1718.565101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_interrupt -r 2 [ 1718.632703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 2 [ 1718.685941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_interrupt -r 2 [ 1718.774714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_interrupt -r 2 [ 1718.846785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_interrupt -r 2 [ 1718.894396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_interrupt -r 2 [ 1718.925917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1718.952616] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1718.965841] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1719.326350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1719.357275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1719.393682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1719.453226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1719.516994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1719.594114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1719.665800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1719.774776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1719.798124] LNet: 87086:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1719.847991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1719.890290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1719.955941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.020400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.120102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.184183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1721.080559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_dropped -r 2 [ 1721.117692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 2 [ 1721.140820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_dropped -r 2 [ 1721.164989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 2 [ 1721.223967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_dropped -r 2 [ 1721.282620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_dropped -r 2 [ 1721.329057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_dropped -r 2 [ 1721.377529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_dropped -r 2 [ 1721.426403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1721.449364] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1721.725897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1721.755389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1721.789363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1721.827636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1721.872108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1721.908855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1721.959284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1721.995478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1722.009368] LNet: 87206:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.033817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1722.103569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1722.173880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.232625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.277601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.310694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1722.860396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_aborted -r 2 [ 1722.890715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 2 [ 1722.919665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_aborted -r 2 [ 1722.954690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 2 [ 1722.985915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_aborted -r 2 [ 1723.013442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_aborted -r 2 [ 1723.046256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_aborted -r 2 [ 1723.079734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_aborted -r 2 [ 1723.107140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1723.129956] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1723.317501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1723.337655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1723.364167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1723.386790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1723.409372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1723.437844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1723.480681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1723.505457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1723.523355] LNet: 87327:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.543962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1723.578651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1723.614577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1723.642305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1723.670703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1723.698948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1724.351677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_no_route -r 2 [ 1724.382741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 2 [ 1724.422887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_no_route -r 2 [ 1724.473349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 2 [ 1724.518141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_no_route -r 2 [ 1724.562651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_no_route -r 2 [ 1724.626294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_no_route -r 2 [ 1724.700512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_no_route -r 2 [ 1724.767930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1725.186624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1725.251238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1725.302948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1725.363059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1725.437452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1725.469670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1725.511579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1725.560943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1725.586878] LNet: 87447:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1725.615623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1725.665317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1725.699000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1725.730529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1725.772658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1725.837859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1726.586114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e local_timeout -r 2 [ 1726.620169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 2 [ 1726.650914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e local_timeout -r 2 [ 1726.691765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 2 [ 1726.729888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e local_timeout -r 2 [ 1726.765655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e local_timeout -r 2 [ 1726.830298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e local_timeout -r 2 [ 1726.886234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e local_timeout -r 2 [ 1726.924614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1726.939249] LNet: There was an unexpected network error while writing to 192.168.201.142: rc = -22 [ 1726.942568] LNet: Skipped 1 previous similar message [ 1727.297406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1727.336713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1727.374494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1727.410836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1727.449260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1727.481753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1727.520967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1727.567632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1727.603574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1727.637626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1727.668593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1727.703903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1727.734766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1727.765620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1728.320652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp -m GET -e remote_dropped -r 2 [ 1728.364790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 2 [ 1728.404328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp -m GET -e remote_dropped -r 2 [ 1728.442766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 2 [ 1728.472988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp -m GET -e remote_dropped -r 2 [ 1728.513968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.142@tcp1 -m GET -e remote_dropped -r 2 [ 1728.561445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp -m GET -e remote_dropped -r 2 [ 1728.600887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.42@tcp1 -d 192.168.201.42@tcp1 -m GET -e remote_dropped -r 2 [ 1728.636702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.42@tcp 192.168.201.142@tcp [ 1728.652043] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 1728.909571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1728.943928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1728.985980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1729.028593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1729.059922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1729.092287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1729.116114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1729.143950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1729.185261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1729.217302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1729.252294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1729.294787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1729.332109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1729.358983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1734.424047] LNet: 87760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.438585] LNetError: 87760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.458949] LNet: Removed LNI 192.168.201.42@tcp [ 1734.473829] LNet: Removed LNI 192.168.201.42@tcp1 [ 1734.960186] Key type .llcrypt unregistered [ 1734.962470] Key type ._llcrypt unregistered [ 1754.456488] Key type ._llcrypt registered [ 1754.460677] Key type .llcrypt registered [ 1756.102704] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:16:17 (1769753777) [ 1756.838340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.856854] alg: No test for adler32 (adler32-zlib) [ 1757.797729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.884562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.913924] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1758.978166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1759.014788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.42@tcp [ 1759.038460] LNetError: 88736:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1759.165695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.42@tcp [ 1759.351404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.375659] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:16:27 (1769753787) [ 1765.562881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.605714] LNet: 89325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.609200] LNetError: 89325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.624086] LNet: Removed LNI 192.168.201.42@tcp [ 1765.645667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.729251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.749381] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1766.551331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1766.574364] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 1767.862790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.42@tcp [ 1767.878580] LNetError: 89591:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.42@tcp to recovery [ 1768.799935] LNet: 1 local NIs in recovery (showing 1): 192.168.201.42@tcp [ 1768.813635] LNet: There was an unexpected network error while writing to 192.168.201.42: rc = -22 [ 1769.217249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.42@tcp [ 1778.184960] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:16:40 (1769753800) [ 1780.172397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg142-server [ 1782.138929] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:16:44 (1769753804) [ 1782.467656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.488204] LNet: 90547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.494306] LNetError: 90547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.501048] LNet: Removed LNI 192.168.201.42@tcp [ 1782.536174] LNet: Removed LNI 192.168.201.42@tcp1 [ 1782.592173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.708368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.742760] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1783.646625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1783.765982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1783.839959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1783.961297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1784.058622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1784.118319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1791.023978] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:16:53 (1769753813) [ 1791.288409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.311495] LNet: 91273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.317477] LNetError: 91273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.322346] LNetError: 91273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1791.348877] LNet: Removed LNI 192.168.201.42@tcp [ 1791.397059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.472896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.506782] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1799.329831] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:17:01 (1769753821) [ 1799.645567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.671679] LNet: 91976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.677780] LNetError: 91976:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.693578] LNet: Removed LNI 192.168.201.42@tcp [ 1799.715626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.765783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.785977] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1807.253264] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:17:09 (1769753829) [ 1808.581861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1810.515163] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:17:12 (1769753832) [ 1811.105953] LNet: 93013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1811.115265] LNetError: 93013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.129463] LNet: Removed LNI 192.168.201.42@tcp [ 1811.700986] Key type .llcrypt unregistered [ 1811.703216] Key type ._llcrypt unregistered [ 1817.531816] Key type ._llcrypt registered [ 1817.535616] Key type .llcrypt registered [ 1817.700909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.719707] alg: No test for adler32 (adler32-zlib) [ 1818.642319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1818.678553] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1826.479317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 1831.894709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.142@tcp [ 1832.393689] LNet: 93426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.407487] LNetError: 93426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.435152] LNet: Removed LNI 192.168.201.42@tcp [ 1832.829666] Key type .llcrypt unregistered [ 1832.839280] Key type ._llcrypt unregistered [ 1844.331525] Key type ._llcrypt registered [ 1844.333304] Key type .llcrypt registered [ 1845.754265] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1847.890744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1849.310376] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:17:51 (1769753871) [ 1849.647263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.664904] alg: No test for adler32 (adler32-zlib) [ 1850.590586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.682133] LNet: 94511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.250446] Key type .llcrypt unregistered [ 1851.253206] Key type ._llcrypt unregistered [ 1857.112892] Key type ._llcrypt registered [ 1857.117352] Key type .llcrypt registered [ 1857.349718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1857.374690] alg: No test for adler32 (adler32-zlib) [ 1858.303648] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1858.308318] LNet: 94763:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1858.313309] LNet: 94763:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1858.349631] LNetError: 94775:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.44@tcp is being used as a gateway but routing feature is not turned on [ 1858.366596] LNetError: 94775:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.44@tcp has gone from up to down [ 1858.861402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.878561] LNet: 94821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.882969] LNetError: 94821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.959853] LNet: Removed LNI 192.168.201.42@tcp [ 1860.014690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.107468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1860.157692] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1860.163948] LNet: 94829:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1860.172306] LNet: 94829:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1860.204263] LNetError: 94829:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.43@tcp is a gateway. Can not delete it [ 1860.215716] LNetError: 94829:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.201.43@tcp is a router. Can not be deleted [ 1860.221664] LNetError: 94825:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.45@tcp is being used as a gateway but routing feature is not turned on [ 1860.232562] LNetError: 94825:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1860.239311] LNetError: 94825:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.45@tcp has gone from up to down [ 1860.257621] LNetError: 94825:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1860.273150] LNet: 94829:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.299225] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:18:09 (1769753889) [ 1869.059421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg142-server [ 1871.115273] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:18:13 (1769753893) [ 1873.807269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg142-server [ 1875.920434] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:18:17 (1769753897) [ 1877.679412] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 1879.584355] Key type lgssc registered [ 1881.028274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1904.592925] Key type lgssc unregistered [ 1904.983453] LNet: 97250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.011763] LNetError: 97250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.025306] LNet: Removed LNI 192.168.201.42@tcp [ 1906.677869] Key type .llcrypt unregistered [ 1906.681430] Key type ._llcrypt unregistered [ 1922.558198] Key type ._llcrypt registered [ 1922.559389] Key type .llcrypt registered [ 1923.863313] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:19:06 (1769753946) [ 1924.242889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.251224] alg: No test for adler32 (adler32-zlib) [ 1925.236724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.311165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1925.356354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1925.388821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1925.431418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1931.091643] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:19:13 (1769753953) [ 1931.282547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.306736] LNet: 98654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.323911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.404417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.439677] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1932.344791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.391477] LNet: 98798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.403430] LNetError: 98798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.432489] LNet: Removed LNI 192.168.201.42@tcp [ 1932.446652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.495655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1932.514948] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1932.875444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.913509] LNet: Removed LNI 192.168.201.42@tcp [ 1932.933459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.009502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1933.054638] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1939.135681] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:19:21 (1769753961) [ 1939.474186] LNet: 99417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.481651] LNet: 99417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1939.490226] LNetError: 99417:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.500663] LNetError: 99417:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1939.529476] LNet: Removed LNI 192.168.201.42@tcp [ 1940.235796] Key type .llcrypt unregistered [ 1940.238428] Key type ._llcrypt unregistered [ 1949.374465] Key type ._llcrypt registered [ 1949.390454] Key type .llcrypt registered [ 1949.641317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1949.663634] alg: No test for adler32 (adler32-zlib) [ 1950.540470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.592634] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 1950.596081] LNet: *** cfs_fail_loc=e003, val=100*** [ 1951.099899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 1951.112592] LNet: *** cfs_fail_loc=e003, val=100*** [ 1951.114339] LNet: Skipped 2 previous similar messages [ 1951.118689] Ping reply received with mapping: [ 1951.120380] found metadata reply NI[192.168.201.42@tcp] [ 1951.121652] 2a c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1951.123618] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1951.125838] 5a 5a 5a 5a 5a 5a 5a 5a [ 1951.216797] LNet: 99811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.220828] LNetError: 99811:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.302672] LNet: Removed LNI 192.168.201.42@tcp [ 1952.902767] Key type .llcrypt unregistered [ 1952.904313] Key type ._llcrypt unregistered [ 1966.121610] Key type ._llcrypt registered [ 1966.123225] Key type .llcrypt registered [ 1967.577695] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:19:49 (1769753989) [ 1967.943416] Key type .llcrypt unregistered [ 1967.946785] Key type ._llcrypt unregistered [ 1975.896129] Key type ._llcrypt registered [ 1975.898196] Key type .llcrypt registered [ 1976.171192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.190879] alg: No test for adler32 (adler32-zlib) [ 1977.343415] LNetError: 100846:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1977.823475] Key type .llcrypt unregistered [ 1977.826665] Key type ._llcrypt unregistered [ 1982.996405] Key type ._llcrypt registered [ 1982.997740] Key type .llcrypt registered [ 1984.273614] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:20:06 (1769754006) [ 1984.878356] Key type .llcrypt unregistered [ 1984.882154] Key type ._llcrypt unregistered [ 1991.719569] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1991.728748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1994.294048] Key type ._llcrypt registered [ 1994.295684] Key type .llcrypt registered [ 1994.546634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1994.565218] alg: No test for adler32 (adler32-zlib) [ 1995.497795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.597218] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2005.257492] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:20:27 (1769754027) [ 2005.363693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.397431] LNet: 102612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.407147] LNetError: 102612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2005.419742] LNet: Removed LNI 10.1.2.3@tcp [ 2005.450514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.127575] Key type .llcrypt unregistered [ 2006.129345] Key type ._llcrypt unregistered [ 2013.115271] Key type ._llcrypt registered [ 2013.117545] Key type .llcrypt registered [ 2013.232270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2013.252485] alg: No test for adler32 (adler32-zlib) [ 2014.183939] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2014.263826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2014.391258] LNet: 102903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.395605] LNetError: 102903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.410867] LNet: Removed LNI 192.168.201.42@tcp [ 2015.104426] Key type .llcrypt unregistered [ 2015.106902] Key type ._llcrypt unregistered [ 2028.434848] Key type ._llcrypt registered [ 2028.437766] Key type .llcrypt registered [ 2029.944621] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:20:51 (1769754051) [ 2030.855352] Key type .llcrypt unregistered [ 2030.857638] Key type ._llcrypt unregistered [ 2038.766482] Key type ._llcrypt registered [ 2038.768880] Key type .llcrypt registered [ 2038.966919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2038.979431] alg: No test for adler32 (adler32-zlib) [ 2046.988712] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:21:08 (1769754068) [ 2047.300440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2047.397200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.480839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.525049] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2048.350640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2048.396404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2048.439628] LNet: Removed LNI 192.168.201.42@tcp [ 2048.742080] LNet: 104640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2049.287821] Key type .llcrypt unregistered [ 2049.291641] Key type ._llcrypt unregistered [ 2062.227600] Key type ._llcrypt registered [ 2062.229865] Key type .llcrypt registered [ 2063.613559] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:21:25 (1769754085) [ 2064.168257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.195269] alg: No test for adler32 (adler32-zlib) [ 2065.249789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.339064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2065.405046] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2066.530295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2066.592762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.42@tcp [ 2066.626403] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2066.630483] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.635894] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 0 [ 2066.645680] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2066.647192] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2066.655925] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.659296] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 0 [ 2066.671759] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2066.683398] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2066.695134] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.705088] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 2116 [ 2066.716238] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2066.720966] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2066.729464] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2066.737311] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2066, 0, 2116 [ 2066.751563] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2075.393342] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:21:37 (1769754097) [ 2075.872729] LNet: 106177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.881746] LNetError: 106177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.899572] LNet: Removed LNI 192.168.201.42@tcp [ 2076.660148] Key type .llcrypt unregistered [ 2076.665937] Key type ._llcrypt unregistered [ 2085.085862] Key type ._llcrypt registered [ 2085.093929] Key type .llcrypt registered [ 2085.511229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2085.533435] alg: No test for adler32 (adler32-zlib) [ 2086.508474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2086.554561] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2095.469170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 2105.818310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.887878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2105.912331] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 2107.474128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2107.681942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2107.697762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2107.713881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2110.230325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2110.248652] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2122.304850] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:22:24 (1769754144) [ 2122.660896] LNet: 107679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.668068] LNetError: 107679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.677127] LNet: Removed LNI 192.168.201.42@tcp [ 2123.679161] LNet: Removed LNI 192.168.201.42@tcp1 [ 2123.682818] LNetError: 107679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2123.694133] LNetError: 107679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2123.719597] LNet: Removed LNI 10.1.2.3@tcp99 [ 2124.602569] Key type .llcrypt unregistered [ 2124.606766] Key type ._llcrypt unregistered [ 2132.417558] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2132.420982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2134.896186] Key type ._llcrypt registered [ 2134.900780] Key type .llcrypt registered [ 2135.174504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2135.187647] alg: No test for adler32 (adler32-zlib) [ 2136.126899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.214834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2136.244090] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2136.785901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2136.802321] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2138.871722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2138.900154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2138.932645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2138.962853] LNetError: 108421:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2139.009485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2139.084676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2139.500888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2139.581940] LNet: 108441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.587433] LNetError: 108441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.592966] LNet: Removed LNI 192.168.201.42@tcp [ 2139.606946] LNet: Removed LNI 10.1.2.3@tcp [ 2140.015396] Key type .llcrypt unregistered [ 2140.017613] Key type ._llcrypt unregistered [ 2151.250988] Key type ._llcrypt registered [ 2151.257295] Key type .llcrypt registered [ 2152.680528] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:22:54 (1769754174) [ 2153.447995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2153.477937] alg: No test for adler32 (adler32-zlib) [ 2154.372755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.412280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.434277] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2155.103533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.42@tcp [ 2155.169346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg142-client@tcp [ 2161.089939] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:23:03 (1769754183) [ 2161.230180] LNet: 109968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.235706] LNetError: 109968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.251076] LNet: Removed LNI 192.168.201.42@tcp [ 2161.988148] Key type .llcrypt unregistered [ 2161.991502] Key type ._llcrypt unregistered [ 2168.378919] Key type ._llcrypt registered [ 2168.385275] Key type .llcrypt registered [ 2168.626074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.666151] alg: No test for adler32 (adler32-zlib) [ 2169.584804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2169.612449] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2169.639060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2176.035688] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:23:18 (1769754198) [ 2177.699144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2179.421497] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:23:21 (1769754201) [ 2180.881991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2182.393277] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:23:24 (1769754204) [ 2182.558325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.588552] LNet: 111390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.605043] LNetError: 111390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.617906] LNet: Removed LNI 192.168.201.42@tcp [ 2182.640887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.689991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.714657] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2183.480491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2183.536575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2189.793398] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:23:32 (1769754212) [ 2189.979169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.015215] LNet: 112097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.024959] LNetError: 112097:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.059355] LNet: Removed LNI 192.168.201.42@tcp [ 2190.124558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.192509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2190.241845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2190.951549] Key type .llcrypt unregistered [ 2190.955333] Key type ._llcrypt unregistered [ 2202.406041] Key type ._llcrypt registered [ 2202.412938] Key type .llcrypt registered [ 2203.811813] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:23:45 (1769754225) [ 2204.205556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.215091] alg: No test for adler32 (adler32-zlib) [ 2205.197822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.304558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.330941] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2206.205297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2206.262577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2212.763288] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:23:55 (1769754235) [ 2212.936881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.961596] LNet: 113648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.970888] LNetError: 113648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.984069] LNet: Removed LNI 192.168.201.42@tcp [ 2213.036875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.088992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2213.113309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2219.183708] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:24:01 (1769754241) [ 2219.267153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.297766] LNet: 114224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.330806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.442944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2219.816152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2219.861205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2219.933466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2220.399441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2220.413808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2220.469535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.493642] LNet: 114265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.513652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.568099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2220.950368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2220.984454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2221.010591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2221.410799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2221.458585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2227.000871] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:24:09 (1769754249) [ 2227.243289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.266653] LNet: 114865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.292227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.368146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2227.412447] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2228.164920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2228.188990] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2229.282658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2229.388789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2229.762858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.804312] LNetError: 115144:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.815145] LNet: Removed LNI 192.168.201.42@tcp [ 2230.818826] LNetError: 115144:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2230.846172] LNet: Removed LNI 192.168.201.42@tcp2 [ 2230.899135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.979906] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2230.989327] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2231.000928] LNet: 115150:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2231.670096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2231.936040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.978501] LNet: 115188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.981342] LNet: 115188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2232.993266] LNet: Removed LNI 192.168.201.42@tcp [ 2232.995802] LNetError: 115188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2233.006726] LNetError: 115188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2233.029418] LNet: Removed LNI 192.168.201.42@tcp2 [ 2233.055662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.152717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.184606] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2234.016546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2234.039525] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2234.917748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2234.951828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2235.027565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2235.262758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.319826] LNet: Removed LNI 192.168.201.42@tcp [ 2237.355602] LNet: Removed LNI 192.168.201.42@tcp2 [ 2237.385183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.488045] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2237.496630] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2237.506271] LNet: 115478:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2238.100834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2238.321648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.340272] LNetError: 115515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.344391] LNetError: 115515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2238.348951] LNet: Removed LNI 192.168.201.42@tcp [ 2239.399781] LNet: Removed LNI 192.168.201.42@tcp2 [ 2239.445890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.526624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2239.548429] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2240.311886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2240.347578] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2241.423529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2241.524662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2241.932676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.974665] LNet: 115795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.982146] LNet: 115795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2243.041185] LNet: Removed LNI 192.168.201.42@tcp [ 2243.060970] LNet: Removed LNI 192.168.201.42@tcp2 [ 2243.081576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.180265] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2243.186128] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2243.195328] LNet: 115801:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2243.824558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2243.992751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.024223] LNet: Removed LNI 192.168.201.42@tcp [ 2245.041389] LNet: Removed LNI 192.168.201.42@tcp2 [ 2245.071405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.164713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.200106] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2246.022690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2246.042152] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2247.052247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2247.124447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2247.191439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2247.435891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.459531] LNetError: 116123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.465832] LNetError: 116123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2248.479341] LNet: Removed LNI 192.168.201.42@tcp [ 2249.510056] LNet: Removed LNI 192.168.201.42@tcp2 [ 2249.548697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2249.629085] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2249.635475] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2249.639947] LNet: 116129:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2250.224798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2250.482464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.503457] LNet: Removed LNI 192.168.201.42@tcp [ 2251.571486] LNet: Removed LNI 192.168.201.42@tcp2 [ 2251.656285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.729450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.767612] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2252.479643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2252.512437] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2253.386388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2253.431466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2253.495753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2253.834729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.879198] LNet: Removed LNI 192.168.201.42@tcp [ 2254.898858] LNet: Removed LNI 192.168.201.42@tcp2 [ 2254.953905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.019696] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2255.027144] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2255.032845] LNet: 116469:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.636850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2255.914750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.991202] LNet: Removed LNI 192.168.201.42@tcp [ 2257.013289] LNet: Removed LNI 192.168.201.42@tcp2 [ 2257.063635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.119264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2257.139083] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2257.829768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.857459] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2258.556360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2258.596128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2258.632617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2258.670591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2259.041437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.060783] LNet: 116822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2259.067101] LNet: 116822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2260.127130] LNet: Removed LNI 192.168.201.42@tcp [ 2261.166106] LNet: Removed LNI 192.168.201.42@tcp2 [ 2261.232536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.309103] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2261.322619] LNet: Added LNI 192.168.201.42@tcp2 [8/256/0/180] [ 2261.919673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2269.556094] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2271.264366] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:24:53 (1769754293) [ 2271.497962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.529989] LNetError: 117587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2271.535567] LNetError: 117587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2271.540819] LNet: Removed LNI 192.168.201.42@tcp [ 2272.620699] LNet: Removed LNI 192.168.201.42@tcp2 [ 2272.711675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.130131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.154435] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2273.872776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2273.899379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2275.006511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2275.041563] LNet: Removed LNI 192.168.201.42@tcp [ 2275.063595] LNet: Removed LNI 10.1.2.3@tcp [ 2275.097269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.194613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2275.217053] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2275.221112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2275.819975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2275.864517] LNet: Removed LNI 192.168.201.42@tcp [ 2275.882290] LNet: Removed LNI 10.1.2.3@tcp [ 2282.494904] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:25:04 (1769754304) [ 2282.741621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.792870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.895351] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2283.000718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2283.009129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2283.016265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2285.590286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.623907] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2286.639191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2286.655551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2289.271766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.42@tcp [ 2289.345558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2289.454363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.42@tcp -m PUT -e local_timeout -r 1 [ 2289.575795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2289.682880] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2289.700302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2289.949024] LNetError: 118527:0:(lib-move.c:4947:lnet_parse()) 192.168.201.42@tcp, src 192.168.201.42@tcp: Dropping PUT (error -108 looking up sender) [ 2291.744718] LNet: Removed LNI 192.168.201.42@tcp [ 2291.773288] LNet: Removed LNI 10.1.2.3@tcp [ 2292.219394] Key type .llcrypt unregistered [ 2292.226219] Key type ._llcrypt unregistered [ 2303.796858] Key type ._llcrypt registered [ 2303.798761] Key type .llcrypt registered [ 2304.963294] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:25:27 (1769754327) [ 2305.387557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.401886] alg: No test for adler32 (adler32-zlib) [ 2306.283164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.623268] LNet: 119834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.206763] Key type .llcrypt unregistered [ 2307.209217] Key type ._llcrypt unregistered [ 2313.122594] Key type ._llcrypt registered [ 2313.124463] Key type .llcrypt registered [ 2313.298597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2313.321537] alg: No test for adler32 (adler32-zlib) [ 2314.252164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2314.287270] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2322.041778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 2322.214560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2322.737290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.142@tcp [ 2322.952627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2322.986254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2323.020943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2323.057652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2323.085330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2323.128053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2323.161861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2323.208448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2323.241384] LNet: 120280:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2323.266625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2323.315829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2323.347459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2323.378294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2323.407788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2323.467577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2324.137486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2324.371689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2324.387794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2324.408186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2324.431228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2324.472598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2324.511167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2324.518573] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 2324.548504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2324.585684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2324.604476] LNet: 120397:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.628305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2324.662557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2324.699205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2324.726587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2324.753238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2324.778976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2325.491694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.142@tcp [ 2325.713121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2325.742568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2325.772792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2325.797863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2325.825750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2325.865402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2325.905521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2325.929656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2325.957801] LNet: 120514:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2325.976693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2326.032305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2326.089674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2326.147750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2326.202912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2326.242652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2326.923378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2327.208597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2327.236941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2327.281836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2327.302433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2327.323762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2327.379056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2327.415882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2327.462676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2327.532593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2327.594990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2327.637551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2327.664979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2327.698487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2327.752052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2328.149543] LNet: 120676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.156585] LNetError: 120676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.175313] LNet: Removed LNI 192.168.201.42@tcp [ 2328.559575] Key type .llcrypt unregistered [ 2328.564678] Key type ._llcrypt unregistered [ 2345.579591] Key type ._llcrypt registered [ 2345.584405] Key type .llcrypt registered [ 2347.041276] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:26:08 (1769754368) [ 2347.510912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.540177] alg: No test for adler32 (adler32-zlib) [ 2348.439050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.839131] LNet: 121560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.545547] Key type .llcrypt unregistered [ 2349.546835] Key type ._llcrypt unregistered [ 2355.299674] Key type ._llcrypt registered [ 2355.301333] Key type .llcrypt registered [ 2355.437239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2355.445917] alg: No test for adler32 (adler32-zlib) [ 2356.301989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2356.324096] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2363.672791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.142@tcp [ 2373.073987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.116830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2373.143063] LNet: Added LNI 192.168.201.42@tcp1 [8/256/0/180] [ 2374.315301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2375.093994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.142@tcp [ 2375.358925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2375.386672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2375.407688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2375.438364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2375.499991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.541110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.596377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.640368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2375.667874] LNet: 122208:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2375.701226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2375.731440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2375.775364] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.142@tcp [ 2375.781147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2375.813893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2375.852605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2375.894669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.618429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2376.846809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2376.897450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2376.935683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2376.979389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2377.035298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2377.085209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2377.159869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2377.207561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2377.221715] LNet: 122326:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.236526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2377.264706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2377.301852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2377.342715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2377.373741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2377.420408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2378.087430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.142@tcp [ 2378.321189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2378.355550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2378.386845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2378.428936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2378.459995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2378.486809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2378.513875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2378.556187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2378.583722] LNet: 122443:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2378.606599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2378.670808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2378.724715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2378.772767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2378.811504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2378.865861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2379.563503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2379.870791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2379.917046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2379.954632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2379.986819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2380.014268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2380.046467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2380.091439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2380.126634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2380.176428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2380.222915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2380.255235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2380.293338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2380.336690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2380.377965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2385.162603] LNet: 122640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.169642] LNetError: 122640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.181692] LNet: Removed LNI 192.168.201.42@tcp [ 2385.198391] LNet: Removed LNI 192.168.201.42@tcp1 [ 2385.755637] Key type .llcrypt unregistered [ 2385.758743] Key type ._llcrypt unregistered [ 2404.511969] Key type ._llcrypt registered [ 2404.513769] Key type .llcrypt registered [ 2405.941171] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:27:07 (1769754427) [ 2412.792934] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:27:15 (1769754435) [ 2419.948349] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:27:22 (1769754442) [ 2426.831544] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:27:29 (1769754449) [ 2434.862976] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:27:36 (1769754456) [ 2444.517713] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:27:45 (1769754465) [ 2453.251780] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:27:54 (1769754474) [ 2461.043396] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:28:03 (1769754483) [ 2469.177594] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:28:11 (1769754491) [ 2476.783875] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:28:18 (1769754498) [ 2483.772954] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:28:25 (1769754505) [ 2492.572494] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:28:34 (1769754514) [ 2499.582312] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:28:41 (1769754521) [ 2506.685893] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:28:48 (1769754528) [ 2514.850903] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:28:56 (1769754536) [ 2523.732635] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:29:05 (1769754545) [ 2531.344994] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:29:13 (1769754553) [ 2538.497773] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:29:20 (1769754560) [ 2539.002473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2539.012255] alg: No test for adler32 (adler32-zlib) [ 2539.908359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2540.009754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2540.027724] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2541.259698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2541.296337] LNet: 133350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2541.315369] LNetError: 133350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2541.346660] LNet: Removed LNI 192.168.201.42@tcp [ 2541.385257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2541.469535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2541.494098] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2547.374335] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:29:29 (1769754569) [ 2554.038227] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:29:36 (1769754576) [ 2562.428594] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:29:44 (1769754584) [ 2570.046519] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:29:52 (1769754592) [ 2576.082354] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:29:58 (1769754598) [ 2583.476833] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:30:05 (1769754605) [ 2591.327593] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:30:13 (1769754613) [ 2598.985726] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:30:20 (1769754620) [ 2607.610139] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:30:29 (1769754629) [ 2616.202257] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:30:38 (1769754638) [ 2623.096618] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:30:45 (1769754645) [ 2629.908868] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:30:52 (1769754652) [ 2637.249382] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2424 sec ========== 01:30:59 (1769754659) [ 2637.660390] LNet: 140812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.665526] LNetError: 140812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.686535] LNet: Removed LNI 192.168.201.42@tcp [ 2638.398382] Key type .llcrypt unregistered [ 2638.401117] Key type ._llcrypt unregistered [ 2649.078735] Key type ._llcrypt registered [ 2649.093976] Key type .llcrypt registered [ 2649.378449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2649.391289] alg: No test for adler32 (adler32-zlib) [ 2650.595731] Lustre: Lustre: Build Version: 2.17.50_72_g285a242 [ 2650.979222] LNet: Added LNI 192.168.201.42@tcp [8/256/0/180] [ 2652.927419] Key type lgssc registered [ 2654.401721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2716.799221] Lustre: Mounted lustre-client [ 2722.059135] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2735.106609] LustreError: 143262:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e0b73380000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2735.130074] LustreError: 143262:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2735.156310] Lustre: Unmounted lustre-client [ 2771.772504] Key type lgssc unregistered [ 2772.190700] LNet: 143745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.199212] LNetError: 143745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.229610] LNet: Removed LNI 192.168.201.42@tcp [ 2772.999505] Key type .llcrypt unregistered [ 2773.005590] Key type ._llcrypt unregistered