[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439117068 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002345] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007595] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012030] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.021210] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026117] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029115] *** VALIDATE tmpfs *** [ 0.031300] *** VALIDATE proc *** [ 0.032235] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034245] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036005] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038021] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.041958] debug: unmapping init [mem 0xffffffffb6c59000-0xffffffffb6c60fff] [ 0.043941] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044501] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045016] ... version: 2 [ 0.045870] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.046760] ... value mask: 0000ffffffffffff [ 0.047011] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.048819] ... event mask: 000000070000000f [ 0.049249] rcu: Hierarchical SRCU implementation. [ 0.051154] smp: Bringing up secondary CPUs ... [ 0.052405] x86: Booting SMP configuration: [ 0.053024] .... node #0, CPUs: #1 #2 #3 [ 0.056493] smp: Brought up 1 node, 4 CPUs [ 0.058013] smpboot: Max logical packages: 1 [ 0.059018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.128873] node 0 deferred pages initialised in 67ms [ 0.132008] devtmpfs: initialized [ 0.133267] x86/mm: Memory block size: 128MB [ 0.135879] gcov: version magic: 0x41383552 [ 0.139165] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143404] pinctrl core: initialized pinctrl subsystem [ 0.145164] [ 0.145669] ************************************************************* [ 0.148014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149010] ** ** [ 0.151025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153009] ** ** [ 0.154010] ** This means that this kernel is built to expose internal ** [ 0.156008] ** IOMMU data structures, which may compromise security on ** [ 0.157008] ** your system. ** [ 0.159037] ** ** [ 0.160000] ** If you see this message and you are not debugging the ** [ 0.160000] ** kernel, report this immediately to your vendor! ** [ 0.162011] ** ** [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165022] ************************************************************* [ 0.168685] NET: Registered protocol family 16 [ 0.170509] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179014] cpuidle: using governor menu [ 0.181520] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184479] PCI: Using configuration type 1 for base access [ 0.186163] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196093] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199085] cryptd: max_cpu_qlen set to 1000 [ 0.200350] ACPI: Added _OSI(Module Device) [ 0.202013] ACPI: Added _OSI(Processor Device) [ 0.203012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208356] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213466] ACPI: Interpreter enabled [ 0.215058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.216044] ACPI: Using IOAPIC for interrupt routing [ 0.218085] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228584] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240024] acpiphp: Slot [2] registered [ 0.240925] acpiphp: Slot [5] registered [ 0.241080] acpiphp: Slot [6] registered [ 0.242108] acpiphp: Slot [3] registered [ 0.243103] acpiphp: Slot [4] registered [ 0.244081] acpiphp: Slot [7] registered [ 0.244945] acpiphp: Slot [8] registered [ 0.246060] acpiphp: Slot [9] registered [ 0.246968] acpiphp: Slot [10] registered [ 0.248074] acpiphp: Slot [11] registered [ 0.249056] acpiphp: Slot [12] registered [ 0.249951] acpiphp: Slot [13] registered [ 0.250096] acpiphp: Slot [14] registered [ 0.251034] acpiphp: Slot [15] registered [ 0.251963] acpiphp: Slot [16] registered [ 0.253091] acpiphp: Slot [17] registered [ 0.254036] acpiphp: Slot [18] registered [ 0.256098] acpiphp: Slot [19] registered [ 0.257125] acpiphp: Slot [20] registered [ 0.258091] acpiphp: Slot [21] registered [ 0.259058] acpiphp: Slot [22] registered [ 0.259964] acpiphp: Slot [23] registered [ 0.260000] acpiphp: Slot [24] registered [ 0.262057] acpiphp: Slot [25] registered [ 0.263026] acpiphp: Slot [26] registered [ 0.263936] acpiphp: Slot [27] registered [ 0.265053] acpiphp: Slot [28] registered [ 0.266095] acpiphp: Slot [29] registered [ 0.268111] acpiphp: Slot [30] registered [ 0.269064] acpiphp: Slot [31] registered [ 0.269888] PCI host bridge to bus 0000:00 [ 0.271020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.272015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.274012] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.275011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.277012] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279123] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280706] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.282806] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.287012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.289337] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.291012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.292014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.294014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.296528] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.298810] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.301035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.302621] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.315431] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.317915] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.321597] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.326022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.331015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.344015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.351640] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.355014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.360018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.370014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.378333] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.380230] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.381292] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.382203] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.384136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.388127] iommu: Default domain type: Passthrough [ 0.389317] SCSI subsystem initialized [ 0.390087] ACPI: bus type USB registered [ 0.390844] usbcore: registered new interface driver usbfs [ 0.392066] usbcore: registered new interface driver hub [ 0.393055] usbcore: registered new device driver usb [ 0.394126] pps_core: LinuxPPS API ver. 1 registered [ 0.395007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.397065] PTP clock support registered [ 0.399061] EDAC MC: Ver: 3.0.0 [ 0.400168] PCI: Using ACPI for IRQ routing [ 0.401586] NetLabel: Initializing [ 0.403008] NetLabel: domain hash size = 128 [ 0.403944] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.405104] NetLabel: unlabeled traffic allowed by default [ 0.408160] vgaarb: loaded [ 0.409447] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.411025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.417000] clocksource: Switched to clocksource kvm-clock [ 0.518866] VFS: Disk quotas dquot_6.6.0 [ 0.520402] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.522700] *** VALIDATE ramfs *** [ 0.523904] *** VALIDATE hugetlbfs *** [ 0.525482] pnp: PnP ACPI init [ 0.527743] pnp: PnP ACPI: found 6 devices [ 0.545394] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.548471] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.550234] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.551768] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.553328] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.555019] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.556959] NET: Registered protocol family 2 [ 0.558624] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.562105] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.564789] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.568439] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.571611] TCP: Hash tables configured (established 65536 bind 65536) [ 0.573756] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.575775] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.577650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.579629] NET: Registered protocol family 1 [ 0.581541] RPC: Registered named UNIX socket transport module. [ 0.582901] RPC: Registered udp transport module. [ 0.583831] RPC: Registered tcp transport module. [ 0.584799] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.586203] NET: Registered protocol family 44 [ 0.587248] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.588555] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.589839] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.591207] PCI: CLS 0 bytes, default 64 [ 0.592776] Unpacking initramfs... [ 1.996613] debug: unmapping init [mem 0xffff9ca97cc64000-0xffff9ca97ffcffff] [ 2.000295] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.002550] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.005117] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.511507] Initialise system trusted keyrings [ 2.512567] Key type blacklist registered [ 2.513531] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.521483] zbud: loaded [ 2.524510] *** VALIDATE nfs *** [ 2.525478] *** VALIDATE nfs4 *** [ 2.526867] pstore: using deflate compression [ 2.529744] Platform Keyring initialized [ 2.629752] NET: Registered protocol family 38 [ 2.631375] Key type asymmetric registered [ 2.632639] Asymmetric key parser 'x509' registered [ 2.634018] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.635941] io scheduler mq-deadline registered [ 2.636908] io scheduler kyber registered [ 2.637941] io scheduler bfq registered [ 2.639124] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.640865] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.642608] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.644368] ACPI: Power Button [PWRF] [ 2.649189] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.654432] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.661546] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.687445] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.720809] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.725835] Non-volatile memory driver v1.3 [ 2.727243] Linux agpgart interface v0.103 [ 2.754687] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.756781] vda: detected capacity change from 0 to 74702848 [ 2.769665] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.771434] vdb: detected capacity change from 0 to 1073741824 [ 2.778340] libphy: Fixed MDIO Bus: probed [ 2.785462] usbcore: registered new interface driver usbserial_generic [ 2.787821] usbserial: USB Serial support registered for generic [ 2.790652] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.794872] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.796747] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.800096] mousedev: PS/2 mouse device common for all mice [ 2.802610] rtc_cmos 00:05: RTC can wake from S4 [ 2.804866] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.807568] rtc_cmos 00:05: registered as rtc0 [ 2.809038] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.810484] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.814569] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.816195] intel_pstate: CPU model not supported [ 2.820932] hid: raw HID events driver (C) Jiri Kosina [ 2.822991] usbcore: registered new interface driver usbhid [ 2.824730] usbhid: USB HID core driver [ 2.825966] drop_monitor: Initializing network drop monitor service [ 2.827819] Initializing XFRM netlink socket [ 2.829430] NET: Registered protocol family 10 [ 2.832164] Segment Routing with IPv6 [ 2.833673] NET: Registered protocol family 17 [ 2.835890] mpls_gso: MPLS GSO support [ 2.840993] RAS: Correctable Errors collector initialized. [ 2.843104] AVX version of gcm_enc/dec engaged. [ 2.844629] AES CTR mode by8 optimization enabled [ 2.918089] sched_clock: Marking stable (2918032557, 0)->(3753014199, -834981642) [ 2.922103] registered taskstats version 1 [ 2.925517] Loading compiled-in X.509 certificates [ 2.927688] zswap: loaded using pool lzo/zbud [ 2.953240] Key type big_key registered [ 2.964972] Key type encrypted registered [ 2.966677] ima: No TPM chip found, activating TPM-bypass! [ 2.970759] ima: Allocated hash algorithm: sha1 [ 2.972506] ima: No architecture policies found [ 2.974406] evm: Initialising EVM extended attributes: [ 2.976338] evm: security.selinux [ 2.977606] evm: security.ima [ 2.978729] evm: security.capability [ 2.980196] evm: HMAC attrs: 0x1 [ 2.982750] rtc_cmos 00:05: setting system clock to 2026-08-17 14:03:10 UTC (1786975390) [ 2.988933] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 2.991490] debug: unmapping init [mem 0xffffffffb6982000-0xffffffffb6c58fff] [ 3.002365] Write protecting the kernel read-only data: 28672k [ 3.004726] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 3.006721] debug: unmapping init [mem 0xffffffffb5914000-0xffffffffb59fffff] [ 3.033122] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.040225] systemd[1]: Detected virtualization kvm. [ 3.041887] systemd[1]: Detected architecture x86-64. [ 3.043784] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.065992] systemd[1]: No hostname configured. [ 3.067167] systemd[1]: Set hostname to . [ 3.068414] random: systemd: uninitialized urandom read (16 bytes read) [ 3.069712] systemd[1]: Initializing machine ID from random generator. [ 3.100620] random: ln: uninitialized urandom read (6 bytes read) [ 3.193407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.196377] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.200890] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.205376] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] [ 3.364722] random: fast init done Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.731916] device-mapper: uevent: version 1.0.3 [ 3.733533] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.420782] virtio_net virtio0 ens2: renamed from eth0 [ 4.436276] scsi host0: ata_piix [ 4.453084] scsi host1: ata_piix [ 4.454731] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.457337] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.165802] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.499436] random: crng init done [ 9.501394] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.842113] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.052816] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.322538] SELinux: Disabled at runtime. [ 11.386338] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.393096] systemd[1]: Detected virtualization kvm. [ 11.394630] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.961477] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.964944] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.969924] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.973796] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.978215] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.988841] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.997638] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Reached target Slices. [ 12.065098] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.526979] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.900060] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.023924] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.044855] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.076221] EDAC sbridge: Ver: 1.1.2 [ 14.393755] Key type dns_resolver registered [ 14.696763] NFS: Registering the id_resolver key type [ 14.698654] Key type id_resolver registered [ 14.700093] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg205-client login: [ 41.658253] libcfs: loading out-of-tree module taints kernel. [ 41.753769] Key type ._llcrypt registered [ 41.765603] Key type .llcrypt registered [ 42.054333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 42.061885] alg: No test for adler32 (adler32-zlib) [ 43.129480] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 43.497899] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 45.135262] Key type lgssc registered [ 46.293301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.160581] hrtimer: interrupt took 4859368 ns [ 183.154773] Lustre: Mounted lustre-client - version 2.17.57_1_gad8061f [ 187.823534] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.596781] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing check_logdir /tmp/testlogs/ [ 203.149407] Lustre: DEBUG MARKER: oleg205-client.virtnet: executing yml_node [ 206.294079] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 208.478904] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 208.863462] Lustre: lustre-OST0000-osc-ffff9ca9d00bf800: disconnect after 23s idle [ 210.650527] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 212.381359] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 17 10:06:38 EDT 2026 [ 228.334118] Lustre: Unmounted lustre-client [ 263.619655] Key type lgssc unregistered [ 263.939971] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.944768] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.968613] LNet: Removed LNI 192.168.202.5@tcp [ 264.572161] Key type .llcrypt unregistered [ 264.577435] Key type ._llcrypt unregistered [ 287.468017] Key type ._llcrypt registered [ 287.469772] Key type .llcrypt registered [ 287.559650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 287.568716] alg: No test for adler32 (adler32-zlib) [ 288.455369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 288.514578] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 288.527596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 291.604064] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 291.792759] LNet: 9299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.795420] LNetError: 9299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.804732] LNet: Removed LNI 192.168.202.5@tcp [ 292.282515] Key type .llcrypt unregistered [ 292.285213] Key type ._llcrypt unregistered [ 298.954519] Key type ._llcrypt registered [ 298.957196] Key type .llcrypt registered [ 300.080260] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:08:06 (1786975686) [ 300.830258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 300.848904] alg: No test for adler32 (adler32-zlib) [ 301.743904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.788327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.799298] LNet: 9713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.810638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.837530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.845437] LNet: 9720:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 309.378868] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:08:15 (1786975695) [ 309.631913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.659739] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.701815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.787910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 316.190787] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:08:22 (1786975702) [ 316.279509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.303833] LNet: 10876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.328383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.452676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 323.281376] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:08:29 (1786975709) [ 323.463969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.519635] LNet: 11462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.568907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.707402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 330.406579] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:08:36 (1786975716) [ 330.488599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.528147] LNet: 12044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.559687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.659136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 330.928041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 336.577860] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:08:42 (1786975722) [ 336.655902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.701316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.790793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 337.102488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 337.619258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 343.176851] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:08:49 (1786975729) [ 343.428773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.445990] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.450197] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 343.471579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.568816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 350.007697] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:08:55 (1786975735) [ 350.128332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.199380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.300484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 350.324202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 350.635348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 350.668134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 351.178633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 351.206628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 351.766190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 351.792613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.151775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 352.190876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.561524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 352.598732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.018333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 353.075798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 353.615131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 353.648047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.037243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 354.070550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 354.463719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 354.518706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 355.283496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 355.350354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 356.014552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 356.057349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 356.459920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 356.516652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.413986] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:09:09 (1786975749) [ 363.542358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.565762] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.569744] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 363.601829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.744758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 363.776617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 370.411743] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:09:16 (1786975756) [ 370.511303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.556327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.655984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 370.682513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 377.027263] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:09:23 (1786975763) [ 377.117875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.161429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.282629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 377.333184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 383.707221] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:09:29 (1786975769) [ 384.048654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.096618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.180887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 384.214867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 390.330389] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:09:36 (1786975776) [ 390.463309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.563647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.679309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 390.707720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 397.436575] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:09:43 (1786975783) [ 397.583777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.620063] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.628325] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 397.666954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.785975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 397.851763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 405.290508] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:09:51 (1786975791) [ 405.492668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.547201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.651939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.039867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 412.182751] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:09:58 (1786975798) [ 412.422989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.455911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.624861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.872688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 420.589709] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:10:06 (1786975806) [ 420.779360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.823984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.934971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.175475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.214407] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:10:15 (1786975815) [ 429.304643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.353987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.986968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.381423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 437.163987] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:10:23 (1786975823) [ 437.271356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.314783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.400116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.439849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.909351] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:10:30 (1786975830) [ 445.106623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.190450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.279849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.342197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.920779] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:10:38 (1786975838) [ 453.079574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.147622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.324809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 460.931137] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:10:46 (1786975846) [ 461.088326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.150369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.323071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 469.198995] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:10:55 (1786975855) [ 469.342437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.387373] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.390866] LNet: 22872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 469.434617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.560952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 470.193242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 478.064968] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:11:04 (1786975864) [ 478.298920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.337299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.497117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 478.559214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 486.357856] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:11:12 (1786975872) [ 486.531393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.605621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.691156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 486.715722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 494.521679] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:11:20 (1786975880) [ 494.649434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.745882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.877534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 494.940449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 501.991692] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:11:27 (1786975887) [ 502.098533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.154969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.270082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 502.298603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 509.751281] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:11:35 (1786975895) [ 509.901450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.955805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.022229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 510.085295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 510.107698] LNetError: 25826:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 510.147776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 510.178639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 516.996167] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:11:43 (1786975903) [ 517.123727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.168098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.288456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qRGRgXRKOA [ 517.375200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qRGRgXRKOA [ 517.461513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.qRGRgXRKOA [ 525.283161] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:11:51 (1786975911) [ 525.516391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.552223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.609521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 525.637476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 525.694715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 533.779975] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:11:59 (1786975919) [ 534.002501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.070198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.177972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 534.217117] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 535.972651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 537.769600] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 539.879277] Key type lgssc registered [ 541.257923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 544.993066] LNetError: 27588:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.5@tcp99 has no available nets [ 548.086562] Key type lgssc unregistered [ 548.440828] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.478045] LNet: Removed LNI 192.168.202.5@tcp [ 550.281792] Key type .llcrypt unregistered [ 550.284479] Key type ._llcrypt unregistered [ 557.691992] Key type ._llcrypt registered [ 557.694379] Key type .llcrypt registered [ 558.833718] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:12:25 (1786975945) [ 559.339408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 559.367582] alg: No test for adler32 (adler32-zlib) [ 560.340791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.439463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 563.547810] LNet: Message forwarding will be enabled in 2s [ 566.623138] LNet: Message forwarding enabled [ 567.115338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 567.176384] LNet: Message forwarding will stop in 2s to 6s [ 571.765847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 572.816803] LNet: Message forwarding disabled [ 574.874947] LNet: Message forwarding will be enabled in 2s [ 577.951330] LNet: Message forwarding enabled [ 582.326601] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:12:48 (1786975968) [ 582.727633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.788326] LNet: 30110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.665313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.721771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 590.498418] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:12:56 (1786975976) [ 590.818253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.856243] LNet: 30691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.882260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.967217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 591.000818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 591.029416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 591.087668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 591.145552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 591.200891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 591.244217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 591.282325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 591.325310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 591.354478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 591.384963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 591.410668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 591.439717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 591.477305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 591.512714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 591.544396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 591.572124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 591.610589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 591.660434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 591.683203] LNetError: 30734:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 598.699957] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:13:04 (1786975984) [ 598.852584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.893342] LNet: 31301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.922548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.988570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.819184] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:13:11 (1786975991) [ 607.702963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg205-server [ 609.466830] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:13:15 (1786975995) [ 609.742772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.767872] LNet: 32237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.794806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.859683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.884861] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 610.732560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[6-10]@tcp [ 610.751730] LNet: 32370:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 610.821264] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 610.853476] LNetError: 32241:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.6@tcp has gone from up to down [ 611.832680] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 611.848547] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 612.841812] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 612.852412] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 614.894773] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 614.940800] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 617.903653] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:13:24 (1786976004) [ 619.004543] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.9@tcp is being used as a gateway but routing feature is not turned on [ 619.023764] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 619.849218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg205-server [ 621.397648] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:13:27 (1786976007) [ 621.637996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.675424] LNet: 33304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.685838] LNetError: 33304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.730851] LNet: Removed LNI 192.168.202.5@tcp [ 621.763664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.807933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.824360] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 622.521194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[6-11/2]@tcp [ 622.562574] LNet: 33423:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 622.568962] LNet: 33423:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 622.592248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[6-11/2]@tcp [ 622.615893] LNetError: 33309:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.8@tcp has gone from up to down [ 622.646816] LNetError: 33309:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 629.238562] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:13:35 (1786976015) [ 629.381147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.422134] LNetError: 34008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.441033] LNet: Removed LNI 192.168.202.5@tcp [ 629.475098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.616951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 629.668716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.697483] LNet: 34029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.701530] LNet: 34029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 629.715711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.775756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.833422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.873838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.923502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 630.013196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.058666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.152740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.261572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.302528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.351794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 630.413358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.456659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.511987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.582983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.610805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.663241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 630.736360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.772161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.829965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.914544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.950855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.005123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.100909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.141920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.212643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.303442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.339754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.401753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 631.462460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.511601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.616708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.473344] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:13:43 (1786976023) [ 639.172205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg205-server [ 640.861573] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:13:47 (1786976027) [ 642.432074] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg205-server [ 644.397549] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:13:50 (1786976030) [ 645.055573] Key type .llcrypt unregistered [ 645.060387] Key type ._llcrypt unregistered [ 653.190614] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 653.467907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 653.489458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 653.504508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 656.372568] Key type ._llcrypt registered [ 656.374932] Key type .llcrypt registered [ 656.652092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.672413] alg: No test for adler32 (adler32-zlib) [ 657.668672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.718615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.751575] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 658.398910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 658.422388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 660.524734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 660.547495] LNet: Removed LNI 10.1.2.3@tcp [ 660.709605] LNet: 36145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.719122] LNetError: 36145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.741730] LNet: Removed LNI 192.168.202.5@tcp [ 661.622861] Key type .llcrypt unregistered [ 661.626695] Key type ._llcrypt unregistered [ 673.343575] Key type ._llcrypt registered [ 673.346338] Key type .llcrypt registered [ 674.814628] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:14:20 (1786976060) [ 675.248856] Key type .llcrypt unregistered [ 675.253984] Key type ._llcrypt unregistered [ 681.972843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 682.156554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 682.169226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 682.178308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 684.863181] Key type ._llcrypt registered [ 684.865333] Key type .llcrypt registered [ 685.175364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.223381] alg: No test for adler32 (adler32-zlib) [ 686.172511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.221728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.249573] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 687.043956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 687.061235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.080247] LNet: 37647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.085295] LNetError: 37647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 690.087801] LNet: Removed LNI 10.1.2.3@tcp [ 690.100811] LNet: Removed LNI 192.168.202.5@tcp [ 690.863215] Key type .llcrypt unregistered [ 690.866983] Key type ._llcrypt unregistered [ 702.757371] Key type ._llcrypt registered [ 702.759953] Key type .llcrypt registered [ 703.921378] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:14:50 (1786976090) [ 704.574394] Key type .llcrypt unregistered [ 704.578805] Key type ._llcrypt unregistered [ 710.436447] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 710.655533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 710.662588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 710.673291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 713.214341] Key type ._llcrypt registered [ 713.220519] Key type .llcrypt registered [ 713.378523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 713.405975] alg: No test for adler32 (adler32-zlib) [ 714.315365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.351908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 714.365985] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 715.922642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 715.947459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.982475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 718.010611] LNet: Removed LNI 10.1.2.3@tcp [ 718.031727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 718.067952] LNet: Removed LNI 10.1.2.31@tcp [ 718.089508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 718.112984] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.822337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 718.849425] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 719.810790] LNet: 39580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.816972] LNetError: 39580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 719.824237] LNet: Removed LNI 10.1.2.31@tcp [ 719.847922] LNet: Removed LNI 10.1.2.3@tcp [ 720.438804] Key type .llcrypt unregistered [ 720.441853] Key type ._llcrypt unregistered [ 732.883339] Key type ._llcrypt registered [ 732.885382] Key type .llcrypt registered [ 734.202403] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:15:20 (1786976120) [ 734.830458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.845969] alg: No test for adler32 (adler32-zlib) [ 735.768356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.821754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.847794] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 736.678764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.697562] LNet: 40538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.713318] LNetError: 40538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.738689] LNet: Removed LNI 192.168.202.5@tcp [ 736.750583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.843770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.5@tcp [ 736.872676] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 737.500020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.5@tcp [ 737.528297] LNet: Removed LNI 192.168.202.5@tcp [ 737.580222] LNet: 40673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.191465] Key type .llcrypt unregistered [ 738.197410] Key type ._llcrypt unregistered [ 749.767690] Key type ._llcrypt registered [ 749.770344] Key type .llcrypt registered [ 751.318144] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:15:37 (1786976137) [ 753.162859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg205-server [ 754.930593] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:15:40 (1786976140) [ 755.616316] Key type .llcrypt unregistered [ 755.620712] Key type ._llcrypt unregistered [ 762.447501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 762.698871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.702539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.710502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 765.394129] Key type ._llcrypt registered [ 765.395972] Key type .llcrypt registered [ 765.590601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 765.610626] alg: No test for adler32 (adler32-zlib) [ 766.655971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.820934] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 767.776610] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 770.329699] LNet: 42519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.341314] LNetError: 42519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.353631] LNet: Removed LNI 192.168.202.5@tcp1 [ 770.375435] LNet: Removed LNI 10.1.2.3@tcp2 [ 771.136869] Key type .llcrypt unregistered [ 771.144090] Key type ._llcrypt unregistered [ 786.378548] Key type ._llcrypt registered [ 786.384598] Key type .llcrypt registered [ 787.906648] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:16:13 (1786976173) [ 788.180123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.209670] alg: No test for adler32 (adler32-zlib) [ 789.140294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.205558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.237126] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 790.254846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.276264] LNet: 43493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.286481] LNetError: 43493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.304322] LNet: Removed LNI 192.168.202.5@tcp [ 790.323285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.410576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 790.457845] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 798.069798] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:16:24 (1786976184) [ 798.164167] LNet: 44075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.176673] LNetError: 44075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.216992] LNet: Removed LNI 192.168.202.5@tcp [ 798.946551] Key type .llcrypt unregistered [ 798.953465] Key type ._llcrypt unregistered [ 806.794955] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.024436] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.037645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.044084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.859892] Key type ._llcrypt registered [ 809.866946] Key type .llcrypt registered [ 810.108751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.124578] alg: No test for adler32 (adler32-zlib) [ 811.132898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.230987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.291079] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 812.236317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.257415] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.722159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.746085] LNet: 44832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.751361] LNetError: 44832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 814.756372] LNet: Removed LNI 10.1.2.3@tcp [ 814.764326] LNet: Removed LNI 192.168.202.5@tcp [ 814.779463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.843328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 814.869575] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 814.877968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.326785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.342781] LNet: 44859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.348338] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 815.354320] LNetError: 44859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 815.358646] LNet: Removed LNI 10.1.2.3@tcp [ 815.369542] LNet: Removed LNI 192.168.202.5@tcp [ 815.387682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.472430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 815.517809] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 815.523690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.202915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 816.222756] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 817.324667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.343394] LNet: 45021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.349406] LNetError: 45021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 817.353903] LNetError: 45021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.358690] LNet: Removed LNI 10.1.2.3@tcp [ 817.360937] LNet: Removed LNI 192.168.202.5@tcp [ 817.368981] LNet: Removed LNI 192.168.202.5@tcp2 [ 817.390541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.457255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 817.480803] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 817.486775] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.501153] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 825.633207] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:16:51 (1786976211) [ 825.739274] LNet: 45611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.749689] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 825.752970] LNetError: 45611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 825.769487] LNet: Removed LNI 10.1.2.3@tcp [ 825.774675] LNet: Removed LNI 192.168.202.5@tcp [ 825.788804] LNet: Removed LNI 192.168.202.5@tcp2 [ 826.452675] Key type .llcrypt unregistered [ 826.455625] Key type ._llcrypt unregistered [ 833.635622] Key type ._llcrypt registered [ 833.637425] Key type .llcrypt registered [ 833.789727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.805888] alg: No test for adler32 (adler32-zlib) [ 834.723521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.795241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.812939] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 835.492214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.507853] LNet: 46017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.513736] LNetError: 46017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.533363] LNet: Removed LNI 192.168.202.5@tcp [ 835.555474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.605427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 835.628713] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 836.022610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.038858] LNet: 46047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.043964] LNetError: 46047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 836.061200] LNet: Removed LNI 192.168.202.5@tcp [ 836.078725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.123817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 836.136644] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 836.555122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 836.602141] LNet: Removed LNI 192.168.202.5@tcp [ 836.624307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.678559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 836.702376] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 837.370070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.518903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.529322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.535852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 840.061112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.114112] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.126094] LNet: 46157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 840.140841] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.148517] LNetError: 46157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.169335] LNet: Removed LNI 192.168.202.5@tcp [ 840.197869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.311300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 840.352913] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 847.556570] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:17:13 (1786976233) [ 847.647445] LNet: 46746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.653642] LNetError: 46746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.677869] LNet: Removed LNI 192.168.202.5@tcp [ 848.331729] Key type .llcrypt unregistered [ 848.332676] Key type ._llcrypt unregistered [ 854.916864] Key type ._llcrypt registered [ 854.919536] Key type .llcrypt registered [ 855.083887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.101811] alg: No test for adler32 (adler32-zlib) [ 856.105569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.176849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.222808] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 857.026286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.038854] LNet: 47157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.050160] LNetError: 47157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.064753] LNet: Removed LNI 192.168.202.5@tcp [ 857.086161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.131565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 857.150552] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 864.285952] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:17:30 (1786976250) [ 864.400029] LNet: 47741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.410442] LNetError: 47741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.441872] LNet: Removed LNI 192.168.202.5@tcp [ 865.237756] Key type .llcrypt unregistered [ 865.242697] Key type ._llcrypt unregistered [ 871.245245] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.431348] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.439925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.449431] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 873.914572] Key type ._llcrypt registered [ 873.917329] Key type .llcrypt registered [ 874.107805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.154979] alg: No test for adler32 (adler32-zlib) [ 875.065582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.106105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 875.124710] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 875.775244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.793365] LNet: 48204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.797249] LNetError: 48204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.805686] LNet: Removed LNI 192.168.202.5@tcp [ 875.828931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.900703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 875.937878] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 882.560610] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:17:48 (1786976268) [ 882.656582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.689851] LNet: 48795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.702529] LNetError: 48795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.719615] LNet: Removed LNI 192.168.202.5@tcp [ 882.734681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.909904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 889.672633] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:17:55 (1786976275) [ 889.998530] LNet: 49385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.719748] Key type .llcrypt unregistered [ 890.725912] Key type ._llcrypt unregistered [ 897.658360] Key type ._llcrypt registered [ 897.659981] Key type .llcrypt registered [ 897.855508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.867494] alg: No test for adler32 (adler32-zlib) [ 898.810302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.893596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.930468] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 900.014853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.035923] LNet: 49786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.048132] LNetError: 49786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.067438] LNet: Removed LNI 192.168.202.5@tcp [ 900.086031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.161137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 900.199281] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 908.276060] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:18:13 (1786976293) [ 908.398224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.448893] LNet: 50376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.464423] LNetError: 50376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.475238] LNet: Removed LNI 192.168.202.5@tcp [ 908.490875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.560990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 916.563974] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:18:22 (1786976302) [ 918.553922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 920.718875] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:18:26 (1786976306) [ 922.426612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 924.785750] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:18:30 (1786976310) [ 926.526840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 928.640275] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:18:34 (1786976314) [ 930.686529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 932.779844] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:18:38 (1786976318) [ 934.592299] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 936.634415] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:18:42 (1786976322) [ 936.769627] LNet: 52425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.625676] Key type .llcrypt unregistered [ 937.627810] Key type ._llcrypt unregistered [ 945.409504] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.643629] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.652285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.664732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 948.271932] Key type ._llcrypt registered [ 948.274442] Key type .llcrypt registered [ 948.447121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.460229] alg: No test for adler32 (adler32-zlib) [ 949.367603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.435824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.460224] LNet: Added LNI 192.168.202.5@tcp [8/512/0/180] [ 949.465673] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 949.468244] LNet: 52753:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 949.586938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.605335] LNet: 52772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.611252] LNetError: 52772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 949.616142] LNet: Removed LNI 10.1.2.3@tcp [ 949.618447] LNetError: 52772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.630905] LNet: Removed LNI 192.168.202.5@tcp [ 949.656854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 949.737819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 949.765255] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 949.868557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.907113] LNet: 52796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.914985] LNetError: 52796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.937888] LNet: Removed LNI 192.168.202.5@tcp [ 949.979532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.072535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.113444] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 950.119637] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.123335] LNet: 52804:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.287495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.317706] LNet: 52823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.324426] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.331290] LNet: Removed LNI 10.1.2.3@tcp [ 950.338260] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.360203] LNet: Removed LNI 192.168.202.5@tcp [ 950.383726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.472104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.489618] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 950.493983] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 950.497354] LNet: 52831:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 950.651544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.673708] LNet: 52857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.679931] LNetError: 52857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 950.684744] LNet: Removed LNI 10.1.2.3@tcp [ 950.686953] LNetError: 52857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.705618] LNet: Removed LNI 192.168.202.5@tcp [ 950.737836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.816352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 950.852151] LNet: Added LNI 192.168.202.5@tcp [8/256/0/182] [ 950.862534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 951.047035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.077801] LNet: 52885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.087436] LNetError: 52885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 951.093472] LNet: Removed LNI 10.1.2.3@tcp [ 951.098890] LNetError: 52885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.122736] LNet: Removed LNI 192.168.202.5@tcp [ 951.145961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.205299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 951.223572] LNet: Added LNI 192.168.202.5@tcp [8/256/0/182] [ 951.227092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 951.230899] LNet: 52893:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 951.317835] LNet: Removed LNI 10.1.2.3@tcp [ 951.331758] LNet: Removed LNI 192.168.202.5@tcp [ 952.278353] Key type .llcrypt unregistered [ 952.280278] Key type ._llcrypt unregistered [ 964.663870] Key type ._llcrypt registered [ 964.667186] Key type .llcrypt registered [ 966.015282] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:19:12 (1786976352) [ 967.423824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 969.034995] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:19:15 (1786976355) [ 969.589554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 969.604614] alg: No test for adler32 (adler32-zlib) [ 970.552106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.610555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.638891] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 971.294278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.5 [ 971.319841] LNet: 54161:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 971.330264] LNet: Removed LNI 192.168.202.5@tcp [ 977.918925] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:19:23 (1786976363) [ 978.079538] LNet: 54730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.088890] LNetError: 54730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.118596] LNet: Removed LNI 192.168.202.5@tcp [ 978.707361] Key type .llcrypt unregistered [ 978.709171] Key type ._llcrypt unregistered [ 984.155048] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 984.370654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 986.922350] Key type ._llcrypt registered [ 986.924442] Key type .llcrypt registered [ 987.008279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.022355] alg: No test for adler32 (adler32-zlib) [ 988.318308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 993.921548] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:19:40 (1786976380) [ 994.274884] LNet: 55649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.289324] LNetError: 55649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.322842] LNet: Removed LNI 10.1.2.3@tcp [ 994.959646] Key type .llcrypt unregistered [ 994.961076] Key type ._llcrypt unregistered [ 1000.714556] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.891141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.898087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.903260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1003.766761] Key type ._llcrypt registered [ 1003.770465] Key type .llcrypt registered [ 1004.010945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1004.027619] alg: No test for adler32 (adler32-zlib) [ 1005.033584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1011.252868] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:19:57 (1786976397) [ 1012.795350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1014.382817] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:20:00 (1786976400) [ 1014.555910] LNet: 56835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.577340] LNetError: 56835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1014.599853] LNet: Removed LNI 10.1.2.3@tcp [ 1014.982660] Key type .llcrypt unregistered [ 1014.984551] Key type ._llcrypt unregistered [ 1021.469576] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1021.921343] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1021.935527] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1021.946590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1023.799230] Key type ._llcrypt registered [ 1023.801421] Key type .llcrypt registered [ 1024.059168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.100377] alg: No test for adler32 (adler32-zlib) [ 1025.157462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1025.251425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.948672] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:20:21 (1786976421) [ 1036.396421] LNet: 58065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.402705] LNetError: 58065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.431812] LNet: Removed LNI 10.1.2.3@tcp [ 1036.996723] Key type .llcrypt unregistered [ 1036.998398] Key type ._llcrypt unregistered [ 1041.901990] Key type ._llcrypt registered [ 1041.903353] Key type .llcrypt registered [ 1042.025391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1042.037540] alg: No test for adler32 (adler32-zlib) [ 1042.916492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1042.935146] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1050.953683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1051.773531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_interrupt -r 1 [ 1051.819937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_interrupt -r 1 [ 1051.856399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1051.879429] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1052.078429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.109273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.141868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.180392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.191534] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1052.220270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.256197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.281994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.319521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.349660] LNet: 58507:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1052.369923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.410345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.439649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.467643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.499638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.558813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.657813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1053.479053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_dropped -r 1 [ 1053.533928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_dropped -r 1 [ 1053.608974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1053.638916] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1053.644879] LNet: Skipped 1 previous similar message [ 1053.950391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.002833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1054.030154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.058708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.089291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.129196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.181903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.241856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.278532] LNet: 58616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.311630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.376680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.461940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.506667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.560690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.653795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.718809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1055.547055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_aborted -r 1 [ 1055.608933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_aborted -r 1 [ 1055.660512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1055.681541] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1056.030615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1056.081891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1056.181563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1056.241508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.287927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.329563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.386209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.433086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.453365] LNet: 58726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.466814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.492932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.545611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.577392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.634231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.700968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.763952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1057.500371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_no_route -r 1 [ 1057.533197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_no_route -r 1 [ 1057.588138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1057.925838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.951806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1057.979958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.033494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.084493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.128206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.166320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.225336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.304282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.348569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.388724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.432858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.474464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.536993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.570575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1059.102995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_timeout -r 1 [ 1059.143864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_timeout -r 1 [ 1059.175257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1059.188965] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1059.193831] LNet: Skipped 1 previous similar message [ 1059.414407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.446969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.478947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.540642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.591046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.626952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.706268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.770403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.811158] LNet: 58944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.819077] LNet: 58944:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1059.846562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.913356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.965870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.043490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.089966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.144550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.184822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.641508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_error -r 1 [ 1060.690826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_error -r 1 [ 1060.741869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1061.038432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.099048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.131108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.156677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.188273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.227438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.265683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.325165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.377178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.402681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.451721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.512856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.575872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.653508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.720473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.245137] LNet: 59099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.251567] LNetError: 59099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.338911] LNet: Removed LNI 192.168.202.5@tcp [ 1064.139556] Key type .llcrypt unregistered [ 1064.144947] Key type ._llcrypt unregistered [ 1083.180591] Key type ._llcrypt registered [ 1083.184517] Key type .llcrypt registered [ 1084.792275] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:21:10 (1786976470) [ 1085.875413] Key type .llcrypt unregistered [ 1085.880878] Key type ._llcrypt unregistered [ 1092.631286] Key type ._llcrypt registered [ 1092.633485] Key type .llcrypt registered [ 1092.886645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1092.900914] alg: No test for adler32 (adler32-zlib) [ 1093.807482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1093.828291] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1100.709888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1111.739536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.791678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1111.829889] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1113.313408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_interrupt -r 1 [ 1113.367236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_interrupt -r 1 [ 1113.404200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_interrupt -r 1 [ 1113.488151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_interrupt -r 1 [ 1113.529950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_interrupt -r 1 [ 1113.565496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_interrupt -r 1 [ 1113.620544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_interrupt -r 1 [ 1113.674441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_interrupt -r 1 [ 1113.705335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1113.721797] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1113.726975] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1113.945255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.979509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.017631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.071331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.116751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.180826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.221890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.283682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.333467] LNet: 60573:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.361478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.390581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.425878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.472590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.500947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.546552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.577189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.188738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_dropped -r 1 [ 1115.252483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_dropped -r 1 [ 1115.297201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_dropped -r 1 [ 1115.351320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_dropped -r 1 [ 1115.413627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_dropped -r 1 [ 1115.463587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_dropped -r 1 [ 1115.515435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_dropped -r 1 [ 1115.565500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_dropped -r 1 [ 1115.610741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1115.642931] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1115.648277] LNet: Skipped 2 previous similar messages [ 1115.891911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.933425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.966935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.004376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.032279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.058902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.097513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.129412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.147405] LNet: 60695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.170925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.213433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.253415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.286079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.331820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.378568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.435828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.057177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_aborted -r 1 [ 1117.097932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_aborted -r 1 [ 1117.130686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_aborted -r 1 [ 1117.161433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_aborted -r 1 [ 1117.192070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_aborted -r 1 [ 1117.226190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_aborted -r 1 [ 1117.268803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_aborted -r 1 [ 1117.297541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_aborted -r 1 [ 1117.329427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1117.350268] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1117.359966] LNet: Skipped 2 previous similar messages [ 1117.588379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.604992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1117.619149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.635584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.649436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.668974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.700520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.726849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.738715] LNet: 60816:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.755223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.799102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.830835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.869424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.909420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.938865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.970920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.522428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_no_route -r 1 [ 1118.581979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_no_route -r 1 [ 1118.633253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_no_route -r 1 [ 1118.676757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_no_route -r 1 [ 1118.756858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_no_route -r 1 [ 1118.799432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_no_route -r 1 [ 1118.854875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_no_route -r 1 [ 1118.906211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_no_route -r 1 [ 1118.970754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1119.300340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.336241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.377795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.402804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.430458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.454395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.487024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.517159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.560370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.586857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.615906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.652215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.680629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.724247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.748657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.388890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_timeout -r 1 [ 1120.427446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_timeout -r 1 [ 1120.458484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_timeout -r 1 [ 1120.491984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_timeout -r 1 [ 1120.534840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_timeout -r 1 [ 1120.584693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_timeout -r 1 [ 1120.626961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_timeout -r 1 [ 1120.671757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_timeout -r 1 [ 1120.699594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1120.712791] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1120.716681] LNet: Skipped 5 previous similar messages [ 1120.912739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.934829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.983744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.022302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.057838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.085793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.119524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.145412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.173958] LNet: 61058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.185781] LNet: 61058:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1121.204608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.235601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.261735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.295707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.336543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.371733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.401736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1121.947719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_error -r 1 [ 1121.985241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_error -r 1 [ 1122.015854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_error -r 1 [ 1122.046924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_error -r 1 [ 1122.079915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_error -r 1 [ 1122.108805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_error -r 1 [ 1122.135870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_error -r 1 [ 1122.165816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_error -r 1 [ 1122.195914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1122.427420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.450854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.478080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.515302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.544777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.569375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.599476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.626226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.655807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.693680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.744906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.801268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.840391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.875573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.925460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.907964] LNet: 61260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.913050] LNetError: 61260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1128.924023] LNet: Removed LNI 192.168.202.5@tcp [ 1128.944612] LNet: Removed LNI 192.168.202.5@tcp1 [ 1129.337630] Key type .llcrypt unregistered [ 1129.339171] Key type ._llcrypt unregistered [ 1144.645331] Key type ._llcrypt registered [ 1144.646852] Key type .llcrypt registered [ 1145.777481] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:22:12 (1786976532) [ 1146.655555] Key type .llcrypt unregistered [ 1146.657363] Key type ._llcrypt unregistered [ 1152.806404] Key type ._llcrypt registered [ 1152.813174] Key type .llcrypt registered [ 1152.995297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.016943] alg: No test for adler32 (adler32-zlib) [ 1153.948549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1153.971621] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1160.527439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1161.345526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_dropped -r 1 [ 1161.421908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_dropped -r 1 [ 1161.503388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1161.527357] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1161.716700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.748616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.784692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.852638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.880482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.919868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.965406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.003407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.021914] LNet: 62513:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.038425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.058988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.087661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.123471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.156289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.217887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.282148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.944173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_error -r 1 [ 1162.982654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_error -r 1 [ 1163.024671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1163.037617] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1163.244771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.287770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.348089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.383639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.418632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.464612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.514552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.539354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.549968] LNet: 62623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.558888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.583521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.606277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.634314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.651705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.674228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.705542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.338815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_timeout -r 1 [ 1164.400344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_timeout -r 1 [ 1164.474413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1164.502360] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1164.783092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.812123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.875813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.930032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.997977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.052486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.119087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.185204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.209520] LNet: 62732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.240807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.303152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.353522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.408432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.487636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.545160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.649229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.056826] LNet: 62777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.062341] LNetError: 62777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.083668] LNet: Removed LNI 192.168.202.5@tcp [ 1166.621759] Key type .llcrypt unregistered [ 1166.628031] Key type ._llcrypt unregistered [ 1186.779581] Key type ._llcrypt registered [ 1186.782137] Key type .llcrypt registered [ 1188.734265] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:22:54 (1786976574) [ 1189.523536] Key type .llcrypt unregistered [ 1189.526243] Key type ._llcrypt unregistered [ 1195.354178] Key type ._llcrypt registered [ 1195.355416] Key type .llcrypt registered [ 1195.437735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.446862] alg: No test for adler32 (adler32-zlib) [ 1196.322119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.351843] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1203.157370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1213.564672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.665085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1213.687087] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1215.628608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_dropped -r 1 [ 1215.694637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e remote_dropped -r 1 [ 1215.771201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_dropped -r 1 [ 1215.864121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e remote_dropped -r 1 [ 1215.934796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e remote_dropped -r 1 [ 1216.067100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e remote_dropped -r 1 [ 1216.191923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e remote_dropped -r 1 [ 1216.280386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e remote_dropped -r 1 [ 1216.355442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1216.427516] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1216.442581] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.105@tcp [ 1216.882834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.934868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1216.999510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.062679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.122233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.201901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.332170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1217.424237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1217.478657] LNet: 64264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1217.505964] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1217.508554] LNet: Skipped 2 previous similar messages [ 1217.531306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1217.622107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.710148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.799897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.866145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.928747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.001539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.066871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.816505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_error -r 1 [ 1218.848409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e remote_error -r 1 [ 1218.916263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_error -r 1 [ 1218.975358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e remote_error -r 1 [ 1219.051639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e remote_error -r 1 [ 1219.130855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e remote_error -r 1 [ 1219.177211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e remote_error -r 1 [ 1219.248859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e remote_error -r 1 [ 1219.326604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1219.373680] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1219.935827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.973664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.006723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.050806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1220.107713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1220.154325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1220.217796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1220.276483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1220.304563] LNet: 64388:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1220.345151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1220.411632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1220.461165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.511877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1220.582729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1220.641726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1220.687103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.755429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.573777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_timeout -r 1 [ 1221.641827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e remote_timeout -r 1 [ 1221.690700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_timeout -r 1 [ 1221.723969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e remote_timeout -r 1 [ 1221.768780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e remote_timeout -r 1 [ 1221.821241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e remote_timeout -r 1 [ 1221.874301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e remote_timeout -r 1 [ 1221.919398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e remote_timeout -r 1 [ 1221.993806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.105@tcp [ 1222.019242] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1222.229554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.260871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.289690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.320726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.352799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.384553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.437485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.496414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.522115] LNet: 64511:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.563625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.630591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.752200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.807482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.838271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.869914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.894870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1222.941082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.406806] LNet: 64593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.416547] LNetError: 64593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.427865] LNet: Removed LNI 192.168.202.5@tcp [ 1228.448058] LNet: Removed LNI 192.168.202.5@tcp1 [ 1229.070561] Key type .llcrypt unregistered [ 1229.072562] Key type ._llcrypt unregistered [ 1249.104213] Key type ._llcrypt registered [ 1249.115390] Key type .llcrypt registered [ 1250.810472] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:23:56 (1786976636) [ 1252.298106] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1254.073214] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:24:00 (1786976640) [ 1255.121636] Key type .llcrypt unregistered [ 1255.124052] Key type ._llcrypt unregistered [ 1261.511218] Key type ._llcrypt registered [ 1261.512967] Key type .llcrypt registered [ 1261.717443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1261.737755] alg: No test for adler32 (adler32-zlib) [ 1262.637571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1262.659313] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1269.722878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1270.488453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e network_timeout -r 1 [ 1270.521440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e network_timeout -r 1 [ 1270.572982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1270.594839] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1270.832374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.856114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.876764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.884766] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1270.900597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.922378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.948202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.970927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.995285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1271.013318] LNet: 66141:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.029785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1271.054772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1271.087715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.112404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.142739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.163223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1271.379595] LNet: 66184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.386690] LNetError: 66184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.406210] LNet: Removed LNI 192.168.202.5@tcp [ 1271.715465] Key type .llcrypt unregistered [ 1271.717095] Key type ._llcrypt unregistered [ 1288.634282] Key type ._llcrypt registered [ 1288.637314] Key type .llcrypt registered [ 1288.873885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.889436] alg: No test for adler32 (adler32-zlib) [ 1289.786831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.808956] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1296.416426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1304.304625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.339787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.369514] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1306.215255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e network_timeout -r 1 [ 1306.285348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e network_timeout -r 1 [ 1306.370284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e network_timeout -r 1 [ 1306.438740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e network_timeout -r 1 [ 1306.508241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e network_timeout -r 1 [ 1306.575989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e network_timeout -r 1 [ 1306.631272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e network_timeout -r 1 [ 1306.665555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e network_timeout -r 1 [ 1306.694920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1306.737262] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1307.113190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.151254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.175467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.204914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.230228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.238038] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1307.240667] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1307.246515] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.105@tcp [ 1307.260272] LNet: Skipped 1 previous similar message [ 1307.275637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.306620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.361212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.400604] LNet: 67066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.438358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.491827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.537501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.563768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.588301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.612843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.684746] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1313.231728] LNet: 67144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.234988] LNetError: 67144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.271210] LNet: Removed LNI 192.168.202.5@tcp [ 1314.274291] LNetError: 67144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.300720] LNet: Removed LNI 192.168.202.5@tcp1 [ 1314.833670] Key type .llcrypt unregistered [ 1314.837441] Key type ._llcrypt unregistered [ 1333.787907] Key type ._llcrypt registered [ 1333.791497] Key type .llcrypt registered [ 1335.331412] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:25:21 (1786976721) [ 1335.833629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1335.870853] alg: No test for adler32 (adler32-zlib) [ 1336.786359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.875611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.915452] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1337.854549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.880055] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1338.984980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 1339.075648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1339.119739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1339.153412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1339.212554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1339.244611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1339.292820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.5@tcp [ 1339.339740] LNetError: 68245:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.5@tcp to recovery [ 1339.935400] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1339.939423] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1341.983624] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1343.342228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1345.056615] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1346.173100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1346.223782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1346.362648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.405664] LNet: 68343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.420486] LNetError: 68343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.434424] LNet: Removed LNI 192.168.202.5@tcp [ 1346.471719] LNet: Removed LNI 192.168.202.5@tcp1 [ 1346.533602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.631220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1346.683096] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1348.462863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.508263] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1349.618588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 1349.685470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.802425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1349.922269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1349.980402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1350.028760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1350.095171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1350.191817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.5@tcp [ 1350.225770] LNetError: 68616:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.5@tcp to recovery [ 1350.687807] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1353.922409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1356.831575] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1356.848283] LNet: Skipped 1 previous similar message [ 1361.893696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.912237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.945373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.359793] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:25:55 (1786976755) [ 1369.645856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.673514] LNet: 69334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.681168] LNetError: 69334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.686179] LNetError: 69334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1369.693339] LNet: Removed LNI 192.168.202.5@tcp [ 1369.729372] LNet: Removed LNI 192.168.202.5@tcp1 [ 1369.810242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.885762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.934682] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1370.790636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.809745] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1371.925937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 1372.016533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1372.079879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1372.155166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1372.231861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1372.292090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1372.355175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.5@tcp --health 0 [ 1372.471251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.959698] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.5@tcp [ 1373.983403] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1373.988429] LNet: Skipped 1 previous similar message [ 1388.285992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.460451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1388.506674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1388.544746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.5@tcp --health 500 [ 1388.690221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1395.494193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1395.550754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.566772] LNet: 69830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.571893] LNetError: 69830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.578996] LNetError: 69830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1396.639154] LNet: Removed LNI 192.168.202.5@tcp [ 1396.652636] LNet: Removed LNI 192.168.202.5@tcp1 [ 1396.729528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.908098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.973189] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1397.801297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.821604] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1398.824796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 1398.886306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.007510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1399.075089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1399.117811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1399.153817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1399.190646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1399.231760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.5@tcp --health 0 [ 1400.929452] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1400.942224] LNet: Skipped 3 previous similar messages [ 1401.776922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1411.160671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1411.198097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1411.249281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1418.777502] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:26:44 (1786976804) [ 1419.301310] LNet: 70878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.304182] LNetError: 70878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.311671] LNetError: 70878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1419.319285] LNet: Removed LNI 192.168.202.5@tcp [ 1419.341964] LNet: Removed LNI 192.168.202.5@tcp1 [ 1420.107958] Key type .llcrypt unregistered [ 1420.110681] Key type ._llcrypt unregistered [ 1426.919791] Key type ._llcrypt registered [ 1426.922689] Key type .llcrypt registered [ 1427.199210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1427.224914] alg: No test for adler32 (adler32-zlib) [ 1428.228807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1428.288375] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1437.593859] Lustre: DEBUG MARKER: Initial discovery [ 1437.815277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.105@tcp [ 1440.110349] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1440.156706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1440.215096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.105@tcp [ 1440.249131] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1440.293074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1440.373818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.105@tcp -r 1 -m GET -l 3 [ 1440.457742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.105@tcp -r 1 -m GET -e local_error [ 1440.489226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.105@tcp -r 1 -m PUT -l 6 [ 1440.615198] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1441.836293] Lustre: DEBUG MARKER: Force oleg205-client.virtnet to discover 192.168.202.105@tcp (in background) [ 1441.916127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.105@tcp [ 1443.420806] Lustre: DEBUG MARKER: Force oleg205-server to discover 192.168.202.5@tcp [ 1445.024522] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1446.008693] Lustre: DEBUG MARKER: Wait for 71639 [ 1447.800671] Lustre: DEBUG MARKER: Finished wait on 71639 [ 1447.834065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1447.857591] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1447.873978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1448.318989] LNet: 72099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.324690] LNetError: 72099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.341066] LNet: Removed LNI 192.168.202.5@tcp [ 1448.769597] Key type .llcrypt unregistered [ 1448.770783] Key type ._llcrypt unregistered [ 1468.776661] Key type ._llcrypt registered [ 1468.779758] Key type .llcrypt registered [ 1469.976438] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:27:36 (1786976856) [ 1470.530219] Key type .llcrypt unregistered [ 1470.533623] Key type ._llcrypt unregistered [ 1476.364137] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1476.479774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1476.484281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1476.487832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1479.134724] Key type ._llcrypt registered [ 1479.144796] Key type .llcrypt registered [ 1479.374867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.386518] alg: No test for adler32 (adler32-zlib) [ 1480.292495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.356389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.373494] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1481.212436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1481.234591] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1488.281900] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:27:54 (1786976874) [ 1488.384172] LNet: 74175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.394861] LNetError: 74175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1488.399064] LNet: Removed LNI 10.1.2.3@tcp [ 1488.422144] LNet: Removed LNI 192.168.202.5@tcp [ 1489.005515] Key type .llcrypt unregistered [ 1489.009675] Key type ._llcrypt unregistered [ 1495.521678] Key type ._llcrypt registered [ 1495.523135] Key type .llcrypt registered [ 1495.779433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.796772] alg: No test for adler32 (adler32-zlib) [ 1496.747947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.842554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1496.878685] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1497.467591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1497.494196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1498.208521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.5@tcp [ 1499.298310] LNetError: Unexpected error -22 connecting to 192.168.202.5@tcp at host 192.168.202.5:988 [ 1504.164960] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:28:10 (1786976890) [ 1504.548100] LNet: 75319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.553662] LNetError: 75319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1504.563164] LNet: Removed LNI 10.1.2.3@tcp [ 1504.589643] LNet: Removed LNI 192.168.202.5@tcp [ 1505.126358] Key type .llcrypt unregistered [ 1505.129571] Key type ._llcrypt unregistered [ 1511.112175] Key type ._llcrypt registered [ 1511.117288] Key type .llcrypt registered [ 1511.307908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1511.324265] alg: No test for adler32 (adler32-zlib) [ 1512.222794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.273388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.297160] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1512.993683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1513.011303] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 1513.846955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.5@tcp1 --nid 192.168.202.5@tcp2 [ 1517.530520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp1 192.168.202.5@tcp2 [ 1517.554649] LNetError: 76158:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.5@tcp1 to 192.168.202.5@tcp2 - Route table may be misconfigured [ 1517.571553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp2 192.168.202.5@tcp1 [ 1523.634302] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:28:29 (1786976909) [ 1523.705535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.729880] LNet: 76723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.737594] LNetError: 76723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.748349] LNet: Removed LNI 192.168.202.5@tcp1 [ 1524.775944] LNet: Removed LNI 192.168.202.5@tcp2 [ 1524.841643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.972219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.017963] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1525.679382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1525.699671] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1526.564908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 1526.632689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.5@tcp [ 1526.674816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp1 192.168.202.5@tcp1 [ 1526.730028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.5@tcp -d 192.168.202.5@tcp -e network_timeout [ 1526.760957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -e network_timeout [ 1526.809644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -e network_timeout [ 1526.857686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -e network_timeout [ 1526.919489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1526.952891] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1526.997243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1527.144586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1527.280957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.967394] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1532.860148] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:28:39 (1786976919) [ 1533.005308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.045407] LNet: 77591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.063871] LNetError: 77591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.068960] LNetError: 77591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1533.075186] LNet: Removed LNI 192.168.202.5@tcp [ 1534.123390] LNet: Removed LNI 192.168.202.5@tcp1 [ 1534.170180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.249724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1534.489628] LNet: 77634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.882342] Key type .llcrypt unregistered [ 1534.883988] Key type ._llcrypt unregistered [ 1546.210709] Key type ._llcrypt registered [ 1546.218161] Key type .llcrypt registered [ 1547.825603] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:28:53 (1786976933) [ 1548.222729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.236731] alg: No test for adler32 (adler32-zlib) [ 1549.181410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.723803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1552.228288] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:28:58 (1786976938) [ 1552.296479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.325976] LNet: 78726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.347330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.452353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.485641] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1553.249654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1553.274293] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1554.018510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1554.066739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp1 [ 1554.101627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp1 [ 1560.848250] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:29:06 (1786976946) [ 1563.046707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg205-server [ 1564.753661] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:29:10 (1786976950) [ 1566.813966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg205-server [ 1568.565740] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:29:14 (1786976954) [ 1570.234188] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg205-server [ 1571.842466] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:29:18 (1786976958) [ 1573.743760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg205-server [ 1575.615463] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:29:21 (1786976961) [ 1577.552852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg205-server [ 1579.523159] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:29:25 (1786976965) [ 1581.740168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg205-server [ 1583.472665] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:29:29 (1786976969) [ 1586.126841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg205-server [ 1587.876807] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:29:34 (1786976974) [ 1589.751644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg205-server [ 1591.341318] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:29:37 (1786976977) [ 1593.569429] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg205-server [ 1595.267923] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:29:41 (1786976981) [ 1595.370365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.391575] LNet: 82605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1595.396458] LNetError: 82605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.400901] LNet: Removed LNI 192.168.202.5@tcp [ 1596.462435] LNet: Removed LNI 192.168.202.5@tcp1 [ 1596.524957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.653747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.697622] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1597.567834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1597.674615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1597.884650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.909855] LNet: 82757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.921601] LNetError: 82757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.934643] LNetError: 82757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1597.971155] LNet: Removed LNI 192.168.202.5@tcp [ 1597.989885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.068610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1598.139407] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1599.477313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1599.699850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.729081] LNetError: 82902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.748154] LNet: Removed LNI 192.168.202.5@tcp [ 1599.786260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.893689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.969038] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1600.464690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1600.558955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1600.836595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.874344] LNet: 83049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.885080] LNet: 83049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1601.961908] LNet: Removed LNI 192.168.202.5@tcp [ 1602.023662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.108516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1602.136252] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1603.188142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1603.417533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.440357] LNetError: 83195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.445374] LNetError: 83195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1603.459143] LNet: Removed LNI 192.168.202.5@tcp [ 1603.477645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.540478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.568475] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1604.328266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1604.418178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1604.635287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.684259] LNet: Removed LNI 192.168.202.5@tcp [ 1605.730833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.834826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1605.869104] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1606.806305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1608.240225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.274317] LNet: 83497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.280368] LNet: 83497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1608.292373] LNetError: 83497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.300717] LNetError: 83497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.318843] LNet: Removed LNI 192.168.202.5@tcp [ 1609.354769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.435369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.468161] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1610.189589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1610.269637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1610.466703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.579211] LNet: Removed LNI 192.168.202.5@tcp [ 1611.643871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.784932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1611.824121] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1612.742730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1613.057281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.128711] LNet: Removed LNI 192.168.202.5@tcp [ 1614.190668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.277868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.314115] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1615.027817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1615.148720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1615.377724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.434241] LNet: Removed LNI 192.168.202.5@tcp [ 1616.475665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.542254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1616.566948] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1617.396253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1617.561975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.579795] LNet: 84083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.583848] LNet: 84083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1617.589330] LNetError: 84083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.594058] LNetError: 84083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1618.661729] LNet: Removed LNI 192.168.202.5@tcp [ 1618.684055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.807729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.865128] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1619.863936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1619.979475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1620.350474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.415772] LNet: Removed LNI 192.168.202.5@tcp [ 1621.453925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.548511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1621.596565] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1622.512785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1623.839353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.871901] LNet: Removed LNI 192.168.202.5@tcp [ 1624.943099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.034964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.073602] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1626.102550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1626.208590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.258975] LNet: Removed LNI 192.168.202.5@tcp [ 1626.292469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.372531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.400487] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1627.368249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1627.454723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.497679] LNet: Removed LNI 192.168.202.5@tcp [ 1627.538957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.590599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1627.683927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.734746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.822798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1627.937555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.027719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.105201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1628.216146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.249603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.319059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1628.369102] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1629.064198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.088789] LNet: Removed LNI 192.168.202.5@tcp [ 1629.101648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.165929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1636.290731] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:30:22 (1786977022) [ 1636.612171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.643097] LNet: 85406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.650906] LNet: 85406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1636.699677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.773993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1636.797089] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/180] [ 1638.156303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.200128] LNetError: 85545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.207645] LNetError: 85545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1638.231077] LNet: Removed LNI 192.168.202.5@tcp231 [ 1638.281709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.338976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1638.365222] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/180] [ 1638.372420] LNet: 85553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.776406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1638.809190] LNet: Removed LNI 192.168.202.5@tcp231 [ 1638.826740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1638.861369] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/180] [ 1640.148224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.172932] LNet: Removed LNI 192.168.202.5@tcp231 [ 1640.211519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.317759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1640.361553] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/0] [ 1640.373077] LNet: 85705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1641.016425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1641.051087] LNet: Removed LNI 192.168.202.5@tcp231 [ 1641.079986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1641.120586] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/0] [ 1642.529797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1642.598769] LNet: Removed LNI 192.168.202.5@tcp231 [ 1642.654708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.747629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1642.777367] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/60] [ 1642.782499] LNet: 85857:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1643.172651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1643.194421] LNet: Removed LNI 192.168.202.5@tcp231 [ 1643.210426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1643.240853] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/60] [ 1644.623403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.681690] LNet: Removed LNI 192.168.202.5@tcp231 [ 1644.719932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.846795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1644.926458] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/180] [ 1644.936060] LNet: 86008:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1645.607408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1645.664382] LNet: Removed LNI 192.168.202.5@tcp231 [ 1645.682793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1645.717323] LNet: Added LNI 192.168.202.5@tcp231 [8/256/0/180] [ 1654.622932] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:30:40 (1786977040) [ 1654.738937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.794632] LNet: Removed LNI 192.168.202.5@tcp231 [ 1654.843082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.949631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1654.983503] LNet: Added LNI 192.168.202.5@tcp232 [8/256/0/180] [ 1655.791806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1655.831608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1655.865760] LNet: Removed LNI 192.168.202.5@tcp232 [ 1655.884951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1655.944582] LNet: Added LNI 192.168.202.5@tcp232 [8/256/0/180] [ 1656.697154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1656.758658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1656.810563] LNet: Removed LNI 192.168.202.5@tcp232 [ 1656.830280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1656.861042] LNet: Added LNI 192.168.202.5@tcp232 [8/256/0/180] [ 1657.886270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1657.938469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1657.967355] LNet: Removed LNI 192.168.202.5@tcp232 [ 1657.996384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1658.052210] LNet: Added LNI 192.168.202.5@tcp232 [8/256/0/180] [ 1658.971145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1659.005360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1659.043203] LNet: Removed LNI 192.168.202.5@tcp232 [ 1659.056996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1666.050089] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:30:52 (1786977052) [ 1667.152659] Key type .llcrypt unregistered [ 1667.161382] Key type ._llcrypt unregistered [ 1674.412667] Key type ._llcrypt registered [ 1674.417089] Key type .llcrypt registered [ 1674.719321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.749573] alg: No test for adler32 (adler32-zlib) [ 1675.811933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.872695] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1684.459970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1693.931199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.975887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1694.016905] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1695.427264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_interrupt -r 2 [ 1695.461716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_interrupt -r 2 [ 1695.502448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_interrupt -r 2 [ 1695.544716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_interrupt -r 2 [ 1695.567279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_interrupt -r 2 [ 1695.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_interrupt -r 2 [ 1695.614069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_interrupt -r 2 [ 1695.641208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_interrupt -r 2 [ 1695.661441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1695.671945] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1695.674589] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1695.945931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.033513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.106412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.157488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.213749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.281101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.322778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.362712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.389568] LNet: 88515:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.409809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.472829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.552444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.651404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.703706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.760765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.457598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_dropped -r 2 [ 1697.500920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_dropped -r 2 [ 1697.553534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_dropped -r 2 [ 1697.623982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_dropped -r 2 [ 1697.695001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_dropped -r 2 [ 1697.784924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_dropped -r 2 [ 1697.830753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_dropped -r 2 [ 1697.882711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_dropped -r 2 [ 1697.939227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1697.958733] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1698.351387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1698.411755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1698.483342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1698.604409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1698.706558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1698.780980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1698.871292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1698.949773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1698.993474] LNet: 88636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1699.019149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1699.082256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1699.125286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1699.168779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1699.217051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1699.287824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1700.231236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_aborted -r 2 [ 1700.312404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_aborted -r 2 [ 1700.365731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_aborted -r 2 [ 1700.467409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_aborted -r 2 [ 1700.576848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_aborted -r 2 [ 1700.633877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_aborted -r 2 [ 1700.715742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_aborted -r 2 [ 1700.789630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_aborted -r 2 [ 1700.862238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1700.913705] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1701.348699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1701.399710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1701.498754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1701.566982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1701.595155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1701.623701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1701.667821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1701.739989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1701.762322] LNet: 88756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.802898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1701.846071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1701.880324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1701.912687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1701.948791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1702.012787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.667629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_no_route -r 2 [ 1702.698368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_no_route -r 2 [ 1702.779677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_no_route -r 2 [ 1702.829504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_no_route -r 2 [ 1702.904445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_no_route -r 2 [ 1702.931573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_no_route -r 2 [ 1702.967458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_no_route -r 2 [ 1702.999531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_no_route -r 2 [ 1703.033725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1703.061956] LNet: There was an unexpected network error while writing to 192.168.202.105: rc = -22 [ 1703.343576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1703.402936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1703.471276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1703.574405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1703.631765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1703.693162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1703.742243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1703.777645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1703.796300] LNet: 88876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.814655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1703.844299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1703.877133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1703.906936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1703.946956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1703.990172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1704.570703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e local_timeout -r 2 [ 1704.606296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e local_timeout -r 2 [ 1704.648088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e local_timeout -r 2 [ 1704.684809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e local_timeout -r 2 [ 1704.715232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e local_timeout -r 2 [ 1704.756204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e local_timeout -r 2 [ 1704.785850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e local_timeout -r 2 [ 1704.828612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e local_timeout -r 2 [ 1704.882943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1705.162055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1705.189959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1705.221989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1705.252274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1705.289696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1705.321900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1705.354948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1705.381758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1705.411875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1705.441828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1705.482457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1705.546261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1705.581309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1705.616664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1706.463664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp -m GET -e remote_dropped -r 2 [ 1706.511804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp -d 192.168.202.105@tcp1 -m GET -e remote_dropped -r 2 [ 1706.562725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp -m GET -e remote_dropped -r 2 [ 1706.613833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp -d 192.168.202.5@tcp1 -m GET -e remote_dropped -r 2 [ 1706.649969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp -m GET -e remote_dropped -r 2 [ 1706.680557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.105@tcp1 -m GET -e remote_dropped -r 2 [ 1706.713795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp -m GET -e remote_dropped -r 2 [ 1706.751514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.5@tcp1 -d 192.168.202.5@tcp1 -m GET -e remote_dropped -r 2 [ 1706.784451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.5@tcp 192.168.202.105@tcp [ 1706.823056] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.105@tcp [ 1707.014595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1707.039318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1707.075612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1707.101268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1707.131421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1707.158873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1707.218864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1707.275618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1707.340157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1707.393417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1707.436778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1707.486602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1707.550864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1707.606866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1713.739620] LNet: 89195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.743773] LNetError: 89195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.754874] LNet: Removed LNI 192.168.202.5@tcp [ 1713.768259] LNet: Removed LNI 192.168.202.5@tcp1 [ 1714.335631] Key type .llcrypt unregistered [ 1714.340060] Key type ._llcrypt unregistered [ 1730.798643] Key type ._llcrypt registered [ 1730.800860] Key type .llcrypt registered [ 1732.155517] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:31:58 (1786977118) [ 1732.463708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.476584] alg: No test for adler32 (adler32-zlib) [ 1733.442429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.508756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.539969] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1734.454700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1734.486680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.5@tcp [ 1734.513815] LNetError: 90175:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.5@tcp to recovery [ 1734.623464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.5@tcp [ 1734.744644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1741.208287] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:32:07 (1786977127) [ 1741.427432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.467665] LNet: 90766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.478558] LNetError: 90766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.503728] LNet: Removed LNI 192.168.202.5@tcp [ 1741.526239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.613339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.655252] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1742.466978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1742.507604] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 1743.960286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.5@tcp [ 1743.978125] LNetError: 91033:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.5@tcp to recovery [ 1744.671389] LNet: 1 local NIs in recovery (showing 1): 192.168.202.5@tcp [ 1744.673994] LNet: There was an unexpected network error while writing to 192.168.202.5: rc = -22 [ 1745.222922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.5@tcp [ 1754.560600] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:32:20 (1786977140) [ 1756.638778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg205-server [ 1758.472816] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:32:24 (1786977144) [ 1760.565834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg205-server [ 1762.813759] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:32:28 (1786977148) [ 1765.247030] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg205-server [ 1767.383814] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:32:33 (1786977153) [ 1767.544757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.587497] LNet: 92657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.595782] LNetError: 92657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.601457] LNet: Removed LNI 192.168.202.5@tcp [ 1767.618103] LNet: Removed LNI 192.168.202.5@tcp1 [ 1767.650217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.768911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.815150] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1768.547097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1768.654692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1768.726882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1768.833788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1768.878524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1768.920464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1775.584784] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:32:41 (1786977161) [ 1775.715646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.740274] LNet: 93383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.744706] LNetError: 93383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.752131] LNetError: 93383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1775.789121] LNet: Removed LNI 192.168.202.5@tcp [ 1775.836889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.915222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.931179] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1783.618594] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:32:49 (1786977169) [ 1783.828989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.867209] LNet: 94089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.886994] LNetError: 94089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.922782] LNet: Removed LNI 192.168.202.5@tcp [ 1783.966910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.052410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.095938] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1791.786590] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:32:57 (1786977177) [ 1793.425962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1795.403601] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:33:01 (1786977181) [ 1796.068693] LNet: 95129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.074336] LNetError: 95129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.088623] LNet: Removed LNI 192.168.202.5@tcp [ 1796.904626] Key type .llcrypt unregistered [ 1796.906863] Key type ._llcrypt unregistered [ 1804.264286] Key type ._llcrypt registered [ 1804.270077] Key type .llcrypt registered [ 1804.543309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.574761] alg: No test for adler32 (adler32-zlib) [ 1805.512980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1805.535516] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1813.453368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 1820.131305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.105@tcp [ 1820.613948] LNet: 95556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.619410] LNetError: 95556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.638771] LNet: Removed LNI 192.168.202.5@tcp [ 1821.036330] Key type .llcrypt unregistered [ 1821.038094] Key type ._llcrypt unregistered [ 1832.829175] Key type ._llcrypt registered [ 1832.831238] Key type .llcrypt registered [ 1834.347780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1835.848349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1837.442887] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:33:43 (1786977223) [ 1839.113231] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1841.020560] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:33:46 (1786977226) [ 1843.172478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg205-server [ 1845.041297] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:33:51 (1786977231) [ 1846.733715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg205-server [ 1848.924406] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:33:54 (1786977234) [ 1850.431757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.552951] alg: No test for adler32 (adler32-zlib) [ 1851.722651] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 1852.106748] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1853.847164] Key type lgssc registered [ 1855.426078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1877.795764] Key type lgssc unregistered [ 1878.077916] LNet: 98752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.084068] LNetError: 98752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.106593] LNet: Removed LNI 192.168.202.5@tcp [ 1878.716810] Key type .llcrypt unregistered [ 1878.721574] Key type ._llcrypt unregistered [ 1893.557771] Key type ._llcrypt registered [ 1893.560276] Key type .llcrypt registered [ 1894.726669] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:34:40 (1786977280) [ 1895.155523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1895.170993] alg: No test for adler32 (adler32-zlib) [ 1896.051806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.114579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1896.143490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1896.186261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1896.225825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1901.311398] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:34:47 (1786977287) [ 1901.470486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.498396] LNet: 100164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.543616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.614704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.652967] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1902.679675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.716381] LNet: 100311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.734222] LNetError: 100311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.772904] LNet: Removed LNI 192.168.202.5@tcp [ 1902.794931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.862223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1902.914956] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1903.528342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.558206] LNetError: 100339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.570794] LNet: Removed LNI 192.168.202.5@tcp [ 1903.586101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.661789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1903.691027] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1910.588904] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:34:56 (1786977296) [ 1910.700515] LNet: 100921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.704791] LNet: 100921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1910.711352] LNetError: 100921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.737731] LNet: Removed LNI 192.168.202.5@tcp [ 1911.258783] Key type .llcrypt unregistered [ 1911.261723] Key type ._llcrypt unregistered [ 1918.013845] Key type ._llcrypt registered [ 1918.018786] Key type .llcrypt registered [ 1918.312210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1918.330517] alg: No test for adler32 (adler32-zlib) [ 1919.276576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.348943] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1919.352625] LNet: *** cfs_fail_loc=e003, val=100*** [ 1920.145756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1920.182621] LNet: *** cfs_fail_loc=e003, val=100*** [ 1920.187598] LNet: Skipped 2 previous similar messages [ 1920.195695] Ping reply received with mapping: [ 1920.201261] found metadata reply NI[192.168.202.5@tcp] [ 1920.204131] 05 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1920.210743] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1920.216962] 5a 5a 5a 5a 5a 5a 5a 5a [ 1920.331648] LNet: 101331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.339659] LNetError: 101331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.394173] LNet: Removed LNI 192.168.202.5@tcp [ 1921.903512] Key type .llcrypt unregistered [ 1921.905860] Key type ._llcrypt unregistered [ 1932.609869] Key type ._llcrypt registered [ 1932.612742] Key type .llcrypt registered [ 1933.785457] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:35:20 (1786977320) [ 1934.155658] Key type .llcrypt unregistered [ 1934.157122] Key type ._llcrypt unregistered [ 1940.929867] Key type ._llcrypt registered [ 1940.932573] Key type .llcrypt registered [ 1941.185742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.205626] alg: No test for adler32 (adler32-zlib) [ 1942.411722] LNetError: 102389:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1942.864360] Key type .llcrypt unregistered [ 1942.869470] Key type ._llcrypt unregistered [ 1947.683855] Key type ._llcrypt registered [ 1947.686222] Key type .llcrypt registered [ 1948.897238] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:35:35 (1786977335) [ 1949.721306] Key type .llcrypt unregistered [ 1949.723919] Key type ._llcrypt unregistered [ 1956.632068] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1956.643195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1959.222694] Key type ._llcrypt registered [ 1959.225264] Key type .llcrypt registered [ 1959.430893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1959.442266] alg: No test for adler32 (adler32-zlib) [ 1960.457259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.548554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1969.750954] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:35:56 (1786977356) [ 1969.859979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.910575] LNet: 104172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.921621] LNetError: 104172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1969.939136] LNet: Removed LNI 10.1.2.3@tcp [ 1970.006659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.675792] Key type .llcrypt unregistered [ 1970.681233] Key type ._llcrypt unregistered [ 1976.389292] Key type ._llcrypt registered [ 1976.390922] Key type .llcrypt registered [ 1976.537505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.551560] alg: No test for adler32 (adler32-zlib) [ 1977.441722] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 1977.507722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 1977.730104] LNet: 104470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.748040] LNetError: 104470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.771985] LNet: Removed LNI 192.168.202.5@tcp [ 1978.410511] Key type .llcrypt unregistered [ 1978.417514] Key type ._llcrypt unregistered [ 1990.002597] Key type ._llcrypt registered [ 1990.016245] Key type .llcrypt registered [ 1991.380331] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:36:17 (1786977377) [ 1992.056445] Key type .llcrypt unregistered [ 1992.058685] Key type ._llcrypt unregistered [ 1998.902791] Key type ._llcrypt registered [ 1998.904176] Key type .llcrypt registered [ 1999.147140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.161344] alg: No test for adler32 (adler32-zlib) [ 2006.931787] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:36:32 (1786977392) [ 2007.243448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.291367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.357509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.377235] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2007.943889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.967389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2007.995981] LNet: Removed LNI 192.168.202.5@tcp [ 2008.287873] LNet: 106227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.633710] Key type .llcrypt unregistered [ 2008.637193] Key type ._llcrypt unregistered [ 2021.677230] Key type ._llcrypt registered [ 2021.678481] Key type .llcrypt registered [ 2023.645358] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:36:48 (1786977408) [ 2024.071627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.080639] alg: No test for adler32 (adler32-zlib) [ 2024.972689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.016523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.032971] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2025.881251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2025.922373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.5@tcp [ 2025.960759] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2025.975525] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2025.985693] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 0 [ 2026.001660] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2026.006539] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2026.012320] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2026.018850] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 0 [ 2026.023686] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2026.026529] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2026.029527] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2026.032185] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 2075 [ 2026.036464] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2026.038772] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2026.042543] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2026.044996] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2025, 0, 2075 [ 2026.054639] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2032.451843] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:36:58 (1786977418) [ 2033.327797] LNet: 107767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.343716] LNetError: 107767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.414087] LNet: Removed LNI 192.168.202.5@tcp [ 2035.445613] Key type .llcrypt unregistered [ 2035.447551] Key type ._llcrypt unregistered [ 2042.491194] Key type ._llcrypt registered [ 2042.494800] Key type .llcrypt registered [ 2042.696800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2042.717404] alg: No test for adler32 (adler32-zlib) [ 2043.652599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2043.677704] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2051.149943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 2060.236705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.312552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2060.353798] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 2061.206455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2061.350842] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2061.361382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2061.371824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2063.748189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2063.808337] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2074.690163] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:37:40 (1786977460) [ 2074.820378] LNet: 109276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.829394] LNetError: 109276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.838765] LNet: Removed LNI 192.168.202.5@tcp [ 2075.871339] LNet: Removed LNI 192.168.202.5@tcp1 [ 2075.873455] LNetError: 109276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2075.885596] LNetError: 109276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2075.916132] LNet: Removed LNI 10.1.2.3@tcp99 [ 2076.713333] Key type .llcrypt unregistered [ 2076.723448] Key type ._llcrypt unregistered [ 2083.058447] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2083.061200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2085.694734] Key type ._llcrypt registered [ 2085.697208] Key type .llcrypt registered [ 2085.906866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2085.933182] alg: No test for adler32 (adler32-zlib) [ 2086.937448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.007370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.038110] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2087.953984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2087.981205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2090.070566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2090.125127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2090.184317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2090.224156] LNetError: 110032:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2090.261168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2090.310428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2090.715285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2090.808616] LNet: 110052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.818907] LNetError: 110052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2090.829912] LNet: Removed LNI 10.1.2.3@tcp [ 2091.879275] LNet: Removed LNI 192.168.202.5@tcp [ 2092.909980] Key type .llcrypt unregistered [ 2092.914990] Key type ._llcrypt unregistered [ 2105.796353] Key type ._llcrypt registered [ 2105.803770] Key type .llcrypt registered [ 2107.199282] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:38:13 (1786977493) [ 2107.584722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2107.598849] alg: No test for adler32 (adler32-zlib) [ 2108.509820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.561189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.584417] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2109.516361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2109.593851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg205-client@tcp [ 2115.988958] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:38:22 (1786977502) [ 2116.061817] LNet: 111582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.066996] LNetError: 111582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.110205] LNet: Removed LNI 192.168.202.5@tcp [ 2116.946765] Key type .llcrypt unregistered [ 2116.948474] Key type ._llcrypt unregistered [ 2124.904355] Key type ._llcrypt registered [ 2124.906418] Key type .llcrypt registered [ 2125.202225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.233496] alg: No test for adler32 (adler32-zlib) [ 2126.261850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2126.293798] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2126.327704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2132.886236] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:38:39 (1786977519) [ 2135.042614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2136.684927] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:38:42 (1786977522) [ 2138.126685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2140.322838] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:38:46 (1786977526) [ 2140.483790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.498623] LNet: 113015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2140.503354] LNetError: 113015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2140.527572] LNet: Removed LNI 192.168.202.5@tcp [ 2140.545692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.611704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.639291] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2141.328786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2141.370268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2148.506937] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:38:54 (1786977534) [ 2148.631079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.694778] LNet: 113718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.704685] LNetError: 113718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.717601] LNet: Removed LNI 192.168.202.5@tcp [ 2148.826222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.896855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2148.939879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2149.616897] Key type .llcrypt unregistered [ 2149.620271] Key type ._llcrypt unregistered [ 2161.604803] Key type ._llcrypt registered [ 2161.608545] Key type .llcrypt registered [ 2163.105650] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:39:08 (1786977548) [ 2163.640816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2163.678669] alg: No test for adler32 (adler32-zlib) [ 2164.643084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.736927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2164.766687] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2165.644762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2165.708676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2172.251916] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:39:18 (1786977558) [ 2172.509623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.532680] LNet: 115286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2172.538260] LNetError: 115286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2172.554560] LNet: Removed LNI 192.168.202.5@tcp [ 2172.580699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.664445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2172.709909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2180.168770] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:39:25 (1786977565) [ 2180.284668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.327418] LNet: 115860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.363164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.448399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2180.876726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2180.910913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2180.975364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2181.446891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2181.487552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2181.607339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.628743] LNet: 115899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.661231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.739969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2182.161266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.234741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2182.314714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2182.903609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2182.954697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2190.401410] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:39:36 (1786977576) [ 2190.702499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.731590] LNet: 116500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.753269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.807413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.829818] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2191.540131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2191.557267] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2192.652181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2192.759840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2193.048793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.079050] LNetError: 116779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.085295] LNet: Removed LNI 192.168.202.5@tcp [ 2193.097431] LNet: Removed LNI 192.168.202.5@tcp2 [ 2193.144285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.225074] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2193.231690] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2193.236552] LNet: 116785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2194.005846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2194.288459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.327878] LNetError: 116822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.332962] LNetError: 116822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2194.341725] LNet: Removed LNI 192.168.202.5@tcp [ 2194.349766] LNet: Removed LNI 192.168.202.5@tcp2 [ 2194.369731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.491313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.550880] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2195.435943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2195.462508] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2196.501616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2196.590393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2196.644404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2196.975563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.991149] LNet: 117107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.994088] LNet: 117107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2197.002179] LNetError: 117107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.011849] LNetError: 117107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2197.017174] LNet: Removed LNI 192.168.202.5@tcp [ 2198.069548] LNet: Removed LNI 192.168.202.5@tcp2 [ 2198.111643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.210613] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2198.233046] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2198.246223] LNet: 117113:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.993606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2199.169728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.189568] LNet: Removed LNI 192.168.202.5@tcp [ 2200.234394] LNet: Removed LNI 192.168.202.5@tcp2 [ 2200.263578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.340329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2200.364683] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2201.338390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2201.360772] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2202.563487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2202.648215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2202.969850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.005333] LNetError: 117430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.017891] LNetError: 117430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2203.029249] LNet: Removed LNI 192.168.202.5@tcp [ 2203.039161] LNet: Removed LNI 192.168.202.5@tcp2 [ 2203.062652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.175751] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2203.181266] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2203.185217] LNet: 117436:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.971281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2204.228512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.279402] LNet: Removed LNI 192.168.202.5@tcp [ 2205.292324] LNet: Removed LNI 192.168.202.5@tcp2 [ 2205.317903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.379669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.411032] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2206.492476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.537276] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2207.512044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2207.556176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2207.607540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2207.849431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.889876] LNet: 117758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.892761] LNet: 117758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2208.927482] LNet: Removed LNI 192.168.202.5@tcp [ 2209.969283] LNet: Removed LNI 192.168.202.5@tcp2 [ 2210.022676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.099808] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2210.103568] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2210.108544] LNet: 117764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2210.655808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2210.885940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.910971] LNet: Removed LNI 192.168.202.5@tcp [ 2211.949928] LNet: Removed LNI 192.168.202.5@tcp2 [ 2211.991919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.064159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.122779] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2213.167568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.196294] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2214.303961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2214.359716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2214.447152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2214.771991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.786916] LNetError: 118098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.791562] LNetError: 118098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2215.839246] LNet: Removed LNI 192.168.202.5@tcp [ 2215.863653] LNet: Removed LNI 192.168.202.5@tcp2 [ 2215.925911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.042896] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2216.048921] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2216.054332] LNet: 118105:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2216.893622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2217.265481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.335183] LNet: Removed LNI 192.168.202.5@tcp [ 2218.354744] LNet: Removed LNI 192.168.202.5@tcp2 [ 2218.421402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.465494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.490240] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2219.242488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2219.263395] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2220.087666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2220.131295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2220.166883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2220.221307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2220.649062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.727809] LNet: Removed LNI 192.168.202.5@tcp [ 2222.757897] LNet: Removed LNI 192.168.202.5@tcp2 [ 2222.806358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.901342] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2222.907861] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2223.605493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2230.831606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2232.409499] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:40:18 (1786977618) [ 2232.615959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.646388] LNet: 119225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.658748] LNet: 119225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2232.666186] LNetError: 119225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.669676] LNetError: 119225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2232.683541] LNet: Removed LNI 192.168.202.5@tcp [ 2233.710847] LNet: Removed LNI 192.168.202.5@tcp2 [ 2233.740457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.188343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.206916] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2234.883987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2234.920359] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2235.674916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.715872] LNet: Removed LNI 10.1.2.3@tcp [ 2235.733861] LNet: Removed LNI 192.168.202.5@tcp [ 2235.758256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.832319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2235.855390] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2235.861769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2236.379745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2236.398871] LNet: Removed LNI 192.168.202.5@tcp [ 2236.410567] LNet: Removed LNI 10.1.2.3@tcp [ 2242.623544] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:40:28 (1786977628) [ 2242.751311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.824390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.917887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.951170] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2243.714247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2243.762988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2243.811623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2243.849991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2243.913188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2243.983108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.033895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.090425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.125473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.182823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.216712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.5@tcp [ 2244.393222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2250.969385] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:40:37 (1786977637) [ 2251.154669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.265191] LNet: Removed LNI 192.168.202.5@tcp [ 2252.306689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.414905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2252.593922] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2252.601282] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2252.603564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2254.958325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.980103] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2255.660162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2255.683562] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2258.175177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2258.241586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2258.343641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.5@tcp -m PUT -e local_timeout -r 1 [ 2258.399918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2258.447314] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2258.454961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2258.779169] LNetError: 120904:0:(lib-move.c:4801:lnet_parse()) 192.168.202.5@tcp, src 192.168.202.5@tcp: Dropping PUT (error -108 looking up sender) [ 2259.487983] LNet: Removed LNI 10.1.2.3@tcp [ 2260.519120] LNet: Removed LNI 192.168.202.5@tcp [ 2260.954822] Key type .llcrypt unregistered [ 2260.959856] Key type ._llcrypt unregistered [ 2272.953948] Key type ._llcrypt registered [ 2272.955783] Key type .llcrypt registered [ 2274.211748] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:41:00 (1786977660) [ 2274.583877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.601960] alg: No test for adler32 (adler32-zlib) [ 2275.558945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.933212] LNet: 122218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2276.645584] Key type .llcrypt unregistered [ 2276.648067] Key type ._llcrypt unregistered [ 2283.828942] Key type ._llcrypt registered [ 2283.832781] Key type .llcrypt registered [ 2284.047803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2284.077552] alg: No test for adler32 (adler32-zlib) [ 2285.048286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2285.092584] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2292.878722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 2293.075466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2293.552787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.105@tcp [ 2293.785457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2293.809927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2293.837846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2293.863632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2293.889196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2293.917672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2293.941678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2293.970921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2293.983893] LNet: 122672:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.004684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2294.032792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2294.061418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2294.086332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2294.122531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2294.150694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2294.758963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2295.026233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.049713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2295.078494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.100894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.126380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.150492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.172360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.193828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.206916] LNet: 122789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.221502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.246629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.271555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.295691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.316783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.327274] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.105@tcp [ 2295.347779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2295.926746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.105@tcp [ 2296.147412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2296.183964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2296.223943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2296.259375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2296.284745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2296.312577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2296.333220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2296.356688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2296.367569] LNet: 122906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2296.382580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2296.417165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2296.440811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2296.464759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2296.491794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2296.523868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2297.185241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2297.442964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2297.474326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2297.493133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2297.524641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2297.551383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2297.580375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2297.620784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2297.658954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2297.698749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2297.745345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2297.770639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2297.802412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2297.837665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2297.875314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2298.162460] LNet: 123074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.167330] LNetError: 123074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.244245] LNet: Removed LNI 192.168.202.5@tcp [ 2299.673652] Key type .llcrypt unregistered [ 2299.677746] Key type ._llcrypt unregistered [ 2317.470589] Key type ._llcrypt registered [ 2317.472090] Key type .llcrypt registered [ 2319.103558] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:41:44 (1786977704) [ 2319.479374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.493771] alg: No test for adler32 (adler32-zlib) [ 2320.476407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.957801] LNet: 123972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.710510] Key type .llcrypt unregistered [ 2321.716382] Key type ._llcrypt unregistered [ 2330.486711] Key type ._llcrypt registered [ 2330.491024] Key type .llcrypt registered [ 2330.821342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2330.863273] alg: No test for adler32 (adler32-zlib) [ 2331.925757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2331.977199] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2341.087424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.105@tcp [ 2350.835268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.889568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2350.913414] LNet: Added LNI 192.168.202.5@tcp1 [8/256/0/180] [ 2351.827663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2352.505889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.105@tcp [ 2352.705755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2352.726364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2352.745846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2352.769081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2352.803665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2352.830828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2352.886398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2352.924886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2352.938336] LNet: 124629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.953802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2352.982986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2353.011968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2353.035967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2353.062899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2353.091419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2353.780930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2354.124412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2354.190564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2354.216470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2354.237352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2354.255676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2354.285881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2354.316875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2354.347865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2354.382975] LNet: 124746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2354.425579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2354.459870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2354.501561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2354.528304] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.105@tcp, 192.168.202.105@tcp1 [ 2354.556096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2354.575947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2354.595879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2355.319502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.105@tcp [ 2355.562996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.586905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.614700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.638724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.677782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.720155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.752748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2355.777784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.807253] LNet: 124863:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.823807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.860877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.905165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.951850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.021702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.083127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.745766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2357.018049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.041952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.063832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.098711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.142344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.180633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2357.232776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.293214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.357754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.387711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.418584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.442666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.476333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.522768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2362.514792] LNet: 125065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.521384] LNetError: 125065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.552285] LNet: Removed LNI 192.168.202.5@tcp [ 2363.555417] LNetError: 125065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.575634] LNet: Removed LNI 192.168.202.5@tcp1 [ 2364.100183] Key type .llcrypt unregistered [ 2364.106495] Key type ._llcrypt unregistered [ 2382.870534] Key type ._llcrypt registered [ 2382.872033] Key type .llcrypt registered [ 2384.421714] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:42:50 (1786977770) [ 2384.802056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2384.824554] alg: No test for adler32 (adler32-zlib) [ 2385.846965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.911539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2385.942047] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2386.643975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2386.660472] LNet: Added LNI 192.168.202.5@tcp2 [8/256/0/180] [ 2387.668789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.5@tcp [ 2387.753963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2388.834153] LNet: Removed LNI 192.168.202.5@tcp [ 2394.782775] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:43:01 (1786977781) [ 2396.600807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2398.243522] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:43:04 (1786977784) [ 2406.540475] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:43:12 (1786977792) [ 2414.246465] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:43:20 (1786977800) [ 2420.482343] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:43:26 (1786977806) [ 2426.880088] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:43:32 (1786977812) [ 2432.902082] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:43:39 (1786977819) [ 2438.910961] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:43:44 (1786977824) [ 2445.576585] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:43:51 (1786977831) [ 2451.774250] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:43:57 (1786977837) [ 2457.376335] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:44:03 (1786977843) [ 2463.498576] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:44:10 (1786977850) [ 2470.144790] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:44:16 (1786977856) [ 2477.419691] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:44:23 (1786977863) [ 2484.506518] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:44:30 (1786977870) [ 2492.025756] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:44:38 (1786977878) [ 2497.954153] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:44:44 (1786977884) [ 2509.113629] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:44:55 (1786977895) [ 2517.746510] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:45:03 (1786977903) [ 2517.981033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2518.055662] LNet: 136757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.067967] LNetError: 136757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2519.078453] LNet: Removed LNI 192.168.202.5@tcp2 [ 2519.137761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2519.399920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2519.426955] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2520.658523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2520.675934] LNet: 136902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.683065] LNetError: 136902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.709043] LNet: Removed LNI 192.168.202.5@tcp [ 2520.726423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2520.789306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2520.812680] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2527.334587] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:45:13 (1786977913) [ 2534.280768] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:45:20 (1786977920) [ 2541.890701] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:45:27 (1786977927) [ 2548.753443] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:45:34 (1786977934) [ 2555.515908] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:45:41 (1786977941) [ 2562.411139] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:45:48 (1786977948) [ 2569.275344] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:45:55 (1786977955) [ 2576.996574] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:46:03 (1786977963) [ 2584.542907] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:46:10 (1786977970) [ 2591.700146] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:46:17 (1786977977) [ 2599.678903] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:46:25 (1786977985) [ 2607.267576] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:46:32 (1786977992) [ 2614.339384] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2400 sec ========== 10:46:40 (1786978000) [ 2614.572571] LNet: 144368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.576658] LNetError: 144368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2614.590010] LNet: Removed LNI 192.168.202.5@tcp [ 2615.265477] Key type .llcrypt unregistered [ 2615.268802] Key type ._llcrypt unregistered [ 2623.556204] Key type ._llcrypt registered [ 2623.560503] Key type .llcrypt registered [ 2623.762513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2623.771337] alg: No test for adler32 (adler32-zlib) [ 2624.789888] Lustre: Lustre: Build Version: 2.17.57_1_gad8061f [ 2624.913125] LNet: Added LNI 192.168.202.5@tcp [8/256/0/180] [ 2626.511992] Key type lgssc registered [ 2627.802727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2687.776088] Lustre: Mounted lustre-client - version 2.17.57_1_gad8061f [ 2692.411486] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2699.736169] Lustre: Unmounted lustre-client [ 2728.400294] Key type lgssc unregistered [ 2728.655972] LNet: 147326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2728.661783] LNetError: 147326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2728.684297] LNet: Removed LNI 192.168.202.5@tcp [ 2729.375155] Key type .llcrypt unregistered [ 2729.377725] Key type ._llcrypt unregistered