[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497444007 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003132] x2apic enabled [ 0.004000] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007483] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010137] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.014059] *** VALIDATE selinux *** [ 0.021750] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025557] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027073] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029128] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031124] *** VALIDATE tmpfs *** [ 0.032416] *** VALIDATE proc *** [ 0.033246] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.035270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037105] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040026] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044144] debug: unmapping init [mem 0xffffffffa2659000-0xffffffffa2660fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048000] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052011] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054013] ... fixed-purpose events: 3 [ 0.055010] ... event mask: 000000070000000f [ 0.056309] rcu: Hierarchical SRCU implementation. [ 0.058492] smp: Bringing up secondary CPUs ... [ 0.059651] x86: Booting SMP configuration: [ 0.060019] .... node #0, CPUs: #1 #2 #3 [ 0.063438] smp: Brought up 1 node, 4 CPUs [ 0.065009] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.106090] node 0 deferred pages initialised in 36ms [ 0.110216] devtmpfs: initialized [ 0.111261] x86/mm: Memory block size: 128MB [ 0.115893] gcov: version magic: 0x41383552 [ 0.118327] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.121081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124280] pinctrl core: initialized pinctrl subsystem [ 0.126188] [ 0.126796] ************************************************************* [ 0.129014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.132012] ** ** [ 0.134013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137015] ** ** [ 0.140013] ** This means that this kernel is built to expose internal ** [ 0.142011] ** IOMMU data structures, which may compromise security on ** [ 0.144014] ** your system. ** [ 0.147012] ** ** [ 0.150015] ** If you see this message and you are not debugging the ** [ 0.152010] ** kernel, report this immediately to your vendor! ** [ 0.155014] ** ** [ 0.157011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160014] ************************************************************* [ 0.163729] NET: Registered protocol family 16 [ 0.165663] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174765] cpuidle: using governor menu [ 0.177009] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179583] PCI: Using configuration type 1 for base access [ 0.182126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190304] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193051] cryptd: max_cpu_qlen set to 1000 [ 0.195118] ACPI: Added _OSI(Module Device) [ 0.196009] ACPI: Added _OSI(Processor Device) [ 0.198011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204562] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211452] ACPI: Interpreter enabled [ 0.213044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.214010] ACPI: Using IOAPIC for interrupt routing [ 0.216139] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.218372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228487] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.243331] acpiphp: Slot [2] registered [ 0.245122] acpiphp: Slot [5] registered [ 0.246097] acpiphp: Slot [6] registered [ 0.247099] acpiphp: Slot [3] registered [ 0.249073] acpiphp: Slot [4] registered [ 0.250115] acpiphp: Slot [7] registered [ 0.252091] acpiphp: Slot [8] registered [ 0.253096] acpiphp: Slot [9] registered [ 0.254124] acpiphp: Slot [10] registered [ 0.256087] acpiphp: Slot [11] registered [ 0.257108] acpiphp: Slot [12] registered [ 0.258090] acpiphp: Slot [13] registered [ 0.260125] acpiphp: Slot [14] registered [ 0.261154] acpiphp: Slot [15] registered [ 0.263068] acpiphp: Slot [16] registered [ 0.264084] acpiphp: Slot [17] registered [ 0.265095] acpiphp: Slot [18] registered [ 0.266095] acpiphp: Slot [19] registered [ 0.268097] acpiphp: Slot [20] registered [ 0.269072] acpiphp: Slot [21] registered [ 0.270077] acpiphp: Slot [22] registered [ 0.271057] acpiphp: Slot [23] registered [ 0.272046] acpiphp: Slot [24] registered [ 0.273007] acpiphp: Slot [25] registered [ 0.274067] acpiphp: Slot [26] registered [ 0.275000] acpiphp: Slot [27] registered [ 0.275117] acpiphp: Slot [28] registered [ 0.276000] acpiphp: Slot [29] registered [ 0.278109] acpiphp: Slot [30] registered [ 0.279140] acpiphp: Slot [31] registered [ 0.281069] PCI host bridge to bus 0000:00 [ 0.282013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.296144] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.297857] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.301041] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.305843] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.309503] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.311014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.312010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.314015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.316510] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.318663] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.323739] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.334015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.338014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.342524] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.351014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.360015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383652] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.395017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.403075] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.422012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.438327] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.441391] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.443360] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.445408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.448258] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.453091] iommu: Default domain type: Passthrough [ 0.454000] SCSI subsystem initialized [ 0.455139] ACPI: bus type USB registered [ 0.456176] usbcore: registered new interface driver usbfs [ 0.458080] usbcore: registered new interface driver hub [ 0.460143] usbcore: registered new device driver usb [ 0.461289] pps_core: LinuxPPS API ver. 1 registered [ 0.463015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.465088] PTP clock support registered [ 0.469093] EDAC MC: Ver: 3.0.0 [ 0.471129] PCI: Using ACPI for IRQ routing [ 0.472704] NetLabel: Initializing [ 0.473012] NetLabel: domain hash size = 128 [ 0.475009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.476076] NetLabel: unlabeled traffic allowed by default [ 0.478097] vgaarb: loaded [ 0.479508] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.480012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.486083] clocksource: Switched to clocksource kvm-clock [ 0.596504] VFS: Disk quotas dquot_6.6.0 [ 0.597641] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.599442] *** VALIDATE ramfs *** [ 0.600316] *** VALIDATE hugetlbfs *** [ 0.601413] pnp: PnP ACPI init [ 0.603366] pnp: PnP ACPI: found 6 devices [ 0.630665] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.634215] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.637494] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.639855] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.642301] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.644934] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.648069] NET: Registered protocol family 2 [ 0.650758] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.655789] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.659634] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.665297] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.668897] TCP: Hash tables configured (established 65536 bind 65536) [ 0.671427] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.675150] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.677918] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.681612] NET: Registered protocol family 1 [ 0.684156] RPC: Registered named UNIX socket transport module. [ 0.685927] RPC: Registered udp transport module. [ 0.687455] RPC: Registered tcp transport module. [ 0.689205] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.692445] NET: Registered protocol family 44 [ 0.694685] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.697631] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.699916] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.702516] PCI: CLS 0 bytes, default 64 [ 0.703967] Unpacking initramfs... [ 2.076943] debug: unmapping init [mem 0xffff99a2fcc64000-0xffff99a2fffcffff] [ 2.080867] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.083122] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.085881] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.591569] Initialise system trusted keyrings [ 2.593388] Key type blacklist registered [ 2.595403] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.605337] zbud: loaded [ 2.608658] *** VALIDATE nfs *** [ 2.610137] *** VALIDATE nfs4 *** [ 2.611816] pstore: using deflate compression [ 2.616249] Platform Keyring initialized [ 2.738296] NET: Registered protocol family 38 [ 2.740026] Key type asymmetric registered [ 2.741777] Asymmetric key parser 'x509' registered [ 2.743733] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.747193] io scheduler mq-deadline registered [ 2.749145] io scheduler kyber registered [ 2.750838] io scheduler bfq registered [ 2.752726] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.755692] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.759866] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.763809] ACPI: Power Button [PWRF] [ 2.770210] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.777176] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.789107] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.830365] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.863311] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.870832] Non-volatile memory driver v1.3 [ 2.873402] Linux agpgart interface v0.103 [ 2.912374] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.915472] vda: detected capacity change from 0 to 76779520 [ 2.930193] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.933261] vdb: detected capacity change from 0 to 1073741824 [ 2.939251] libphy: Fixed MDIO Bus: probed [ 2.949296] usbcore: registered new interface driver usbserial_generic [ 2.951951] usbserial: USB Serial support registered for generic [ 2.954346] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.958538] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.960616] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.962881] mousedev: PS/2 mouse device common for all mice [ 2.965550] rtc_cmos 00:05: RTC can wake from S4 [ 2.969253] rtc_cmos 00:05: registered as rtc0 [ 2.971257] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.974460] intel_pstate: CPU model not supported [ 2.974937] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.979919] hid: raw HID events driver (C) Jiri Kosina [ 2.980039] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.981341] usbcore: registered new interface driver usbhid [ 2.985838] usbhid: USB HID core driver [ 2.987482] drop_monitor: Initializing network drop monitor service [ 2.990138] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.990686] Initializing XFRM netlink socket [ 2.996327] NET: Registered protocol family 10 [ 3.000241] Segment Routing with IPv6 [ 3.001564] NET: Registered protocol family 17 [ 3.003861] mpls_gso: MPLS GSO support [ 3.009443] RAS: Correctable Errors collector initialized. [ 3.011243] AVX version of gcm_enc/dec engaged. [ 3.012545] AES CTR mode by8 optimization enabled [ 3.095708] sched_clock: Marking stable (3095535298, 0)->(4006732304, -911197006) [ 3.102531] registered taskstats version 1 [ 3.104623] Loading compiled-in X.509 certificates [ 3.106763] zswap: loaded using pool lzo/zbud [ 3.135300] Key type big_key registered [ 3.147798] Key type encrypted registered [ 3.149404] ima: No TPM chip found, activating TPM-bypass! [ 3.151329] ima: Allocated hash algorithm: sha1 [ 3.152981] ima: No architecture policies found [ 3.154455] evm: Initialising EVM extended attributes: [ 3.155828] evm: security.selinux [ 3.156657] evm: security.ima [ 3.157414] evm: security.capability [ 3.158685] evm: HMAC attrs: 0x1 [ 3.161077] rtc_cmos 00:05: setting system clock to 2026-09-16 17:41:37 UTC (1789580497) [ 3.167474] debug: unmapping init [mem 0xffffffffa3603000-0xffffffffa37fffff] [ 3.170764] debug: unmapping init [mem 0xffffffffa2382000-0xffffffffa2658fff] [ 3.181228] Write protecting the kernel read-only data: 28672k [ 3.184615] debug: unmapping init [mem 0xffffffffa0a03000-0xffffffffa0bfffff] [ 3.188051] debug: unmapping init [mem 0xffffffffa1314000-0xffffffffa13fffff] [ 3.220113] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.227399] systemd[1]: Detected virtualization kvm. [ 3.228567] systemd[1]: Detected architecture x86-64. [ 3.229750] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.251095] systemd[1]: No hostname configured. [ 3.252241] systemd[1]: Set hostname to . [ 3.254184] random: systemd: uninitialized urandom read (16 bytes read) [ 3.255659] systemd[1]: Initializing machine ID from random generator. [ 3.290425] random: ln: uninitialized urandom read (6 bytes read) [ 3.392174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.394411] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.401070] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.404760] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.021536] device-mapper: uevent: version 1.0.3 [ 4.025209] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.751637] virtio_net virtio0 ens2: renamed from eth0 [ 4.791933] scsi host0: ata_piix [ 4.824213] scsi host1: ata_piix [ 4.826052] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.828636] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.158497] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.664394] random: crng init done [ 9.666058] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.090724] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.297824] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.574071] SELinux: Disabled at runtime. [ 11.635732] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.644505] systemd[1]: Detected virtualization kvm. [ 11.646769] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.229897] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.234542] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.243220] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.248358] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.251141] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.260900] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.271966] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK [0[ 12.313720] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.804601] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.167213] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.213949] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.340516] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.359496] EDAC sbridge: Ver: 1.1.2 [ 14.611656] Key type dns_resolver registered [ 14.934850] NFS: Registering the id_resolver key type [ 14.936891] Key type id_resolver registered [ 14.938498] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg438-client login: [ 56.377482] libcfs: loading out-of-tree module taints kernel. [ 56.467427] Key type ._llcrypt registered [ 56.469522] Key type .llcrypt registered [ 56.972290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 56.984989] alg: No test for adler32 (adler32-zlib) [ 58.651696] Lustre: Lustre: Build Version: 2.17.58_39_g2c5a7bf [ 59.832421] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 61.640217] Key type lgssc registered [ 63.447252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.448279] hrtimer: interrupt took 3269829 ns [ 234.587365] Lustre: Mounted lustre-client - version 2.17.58_39_g2c5a7bf [ 239.664434] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.257837] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing check_logdir /tmp/testlogs/ [ 255.908225] Lustre: DEBUG MARKER: oleg438-client.virtnet: executing yml_node [ 260.072629] Lustre: lustre-OST0000-osc-ffff99a3466cc800: disconnect after 23s idle [ 260.784715] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 263.178851] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 265.391655] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 267.081746] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 16 13:46:00 EDT 2026 [ 283.620599] Lustre: Unmounted lustre-client [ 347.425927] Key type lgssc unregistered [ 347.716897] LNet: 9220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.736651] LNetError: 9220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.766477] LNet: Removed LNI 192.168.204.38@tcp [ 348.832172] Key type .llcrypt unregistered [ 348.835579] Key type ._llcrypt unregistered [ 382.257837] Key type ._llcrypt registered [ 382.261445] Key type .llcrypt registered [ 382.458539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 382.485103] alg: No test for adler32 (adler32-zlib) [ 383.396524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 383.441370] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 383.489974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 388.406637] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 388.517553] LNet: 10140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.527560] LNetError: 10140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.550152] LNet: Removed LNI 192.168.204.38@tcp [ 389.250386] Key type .llcrypt unregistered [ 389.252750] Key type ._llcrypt unregistered [ 396.692554] Key type ._llcrypt registered [ 396.696540] Key type .llcrypt registered [ 398.188399] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:48:10 (1789580890) [ 398.632462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 398.661976] alg: No test for adler32 (adler32-zlib) [ 399.612816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.697337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.719265] LNet: 10562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.742480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.795503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 399.810401] LNet: 10569:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 406.970966] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:48:19 (1789580899) [ 407.151211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.183788] LNet: 11151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.224305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.364719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 414.354612] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:48:27 (1789580907) [ 414.494045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.541889] LNet: 11746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.573633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.691988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 421.340969] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:48:34 (1789580914) [ 421.446359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.510908] LNet: 12336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.534892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.642812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 428.227892] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:48:41 (1789580921) [ 428.367413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.404400] LNet: 12936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.429381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.541722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 429.007694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 436.948044] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:48:49 (1789580929) [ 437.071176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.116563] LNet: 13526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.155269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.327654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 437.948199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 438.542273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 446.464421] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:48:58 (1789580938) [ 446.682501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.762571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.866687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 454.776579] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:49:07 (1789580947) [ 454.937814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.001209] LNet: 14724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.015278] LNet: 14724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 455.071638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.237544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 455.312437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 455.860193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 455.920430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 456.502235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 456.533053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 456.851428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 456.904414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 457.277581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 457.303318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 457.523932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 457.588032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 457.928430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 458.011733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 458.650896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 458.723351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 459.192856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 459.215939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 459.565728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 459.602766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 459.846247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 459.875373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 460.183123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 460.248537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 460.512289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 460.541222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 467.360225] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:49:20 (1789580960) [ 467.519046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.574676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.656701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 467.683194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 475.925673] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:49:28 (1789580968) [ 476.205454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.273223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.468931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 476.558802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 484.254716] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:49:36 (1789580976) [ 484.445553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.503590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.586778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 484.639728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 492.139558] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:49:44 (1789580984) [ 492.368321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.385694] LNet: 17206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.390572] LNet: 17206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 492.408957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.474422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 492.536648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 500.504183] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:49:52 (1789580992) [ 500.651294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.723466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.862324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 500.914388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 508.146511] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:50:01 (1789581001) [ 508.455709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.496792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.585488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 508.619305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 515.736221] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:50:08 (1789581008) [ 515.869483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.943142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.039880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.474561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.564439] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:50:15 (1789581015) [ 522.833029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.859797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.165812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 523.575275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.931497] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:50:23 (1789581023) [ 530.162494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.219793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.372519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.679628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.814318] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:50:31 (1789581031) [ 538.947244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.012582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.460488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.648253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 545.358851] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:50:38 (1789581038) [ 545.590433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.642256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.721082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.808489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 553.164028] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:50:46 (1789581046) [ 553.452497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.528888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.632368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 553.732663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 562.865862] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:50:55 (1789581055) [ 563.024030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.076556] LNet: 22712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.081885] LNet: 22712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 563.138975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.364419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 574.616699] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:51:06 (1789581066) [ 574.752350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.832740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.980917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 582.808260] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:51:15 (1789581075) [ 582.901519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.937655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.068176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 583.511618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 590.598977] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:51:23 (1789581083) [ 590.860915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.975541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.110857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 591.190852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 598.668677] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:51:31 (1789581091) [ 598.711071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.747728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.827066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 598.863452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 606.346808] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:51:38 (1789581098) [ 606.992287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.068429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.180406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 607.228287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 616.204970] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:51:48 (1789581108) [ 616.560149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.638376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.794138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 616.872878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 627.487772] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:51:59 (1789581119) [ 627.889380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.974718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.163580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 628.306957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 628.325495] LNetError: 26895:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 628.456545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 628.588910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 639.397349] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:52:11 (1789581131) [ 639.619709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.710289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.946392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rX2fkcE25y [ 640.264912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rX2fkcE25y [ 640.323864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rX2fkcE25y [ 648.829413] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:52:21 (1789581141) [ 649.030901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.113061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.238496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 649.305830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 649.396139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 656.696447] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 13:52:29 (1789581149) [ 656.810967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.863575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.952285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.993918] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 658.472845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 660.061447] Lustre: Lustre: Build Version: 2.17.58_39_g2c5a7bf [ 662.184698] Key type lgssc registered [ 663.494370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 668.069945] LNetError: 28678:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.38@tcp99 has no available nets [ 670.898474] Key type lgssc unregistered [ 671.271934] LNetError: 29815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.296716] LNet: Removed LNI 192.168.204.38@tcp [ 671.990340] Key type .llcrypt unregistered [ 671.992966] Key type ._llcrypt unregistered [ 680.471629] Key type ._llcrypt registered [ 680.476717] Key type .llcrypt registered [ 682.108358] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:52:54 (1789581174) [ 682.777563] Key type .llcrypt unregistered [ 682.779798] Key type ._llcrypt unregistered [ 688.966970] Key type ._llcrypt registered [ 688.970099] Key type .llcrypt registered [ 689.094289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.113523] alg: No test for adler32 (adler32-zlib) [ 690.066826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.136586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 690.152748] LNet: 30735:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 690.157894] LNet: 30735:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 690.164638] LNet: 30735:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 690.179576] LNet: Message forwarding will be enabled in 2s [ 693.280367] LNet: Message forwarding enabled [ 693.712463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 693.750542] LNet: Message forwarding will stop in 2s to 6s [ 697.380617] LNet: Message forwarding disabled [ 698.456607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 698.475521] LNet: 30833:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 698.480481] LNet: 30833:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 698.485371] LNet: 30833:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 698.500822] LNet: Message forwarding will be enabled in 2s [ 698.660548] LNet: 30846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.405562] Key type .llcrypt unregistered [ 699.408988] Key type ._llcrypt unregistered [ 712.689693] Key type ._llcrypt registered [ 712.695207] Key type .llcrypt registered [ 714.045205] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:53:26 (1789581206) [ 714.518605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.536251] alg: No test for adler32 (adler32-zlib) [ 715.472736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.587847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 722.554540] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:53:34 (1789581214) [ 722.665942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.706314] LNet: 32258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.754054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.843670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 722.892930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 722.921606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 722.987627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 723.040454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 723.089872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 723.122521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 723.196207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 723.237190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 723.280481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 723.319598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 723.369309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 723.415589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 723.448190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 723.481719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 723.515306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 723.557278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 723.585927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 723.632771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 723.664471] LNetError: 32308:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 731.107540] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:53:43 (1789581223) [ 731.424602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.455355] LNet: 32888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.478490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.582060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.189595] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:53:52 (1789581232) [ 742.321458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg438-server [ 744.849995] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:53:57 (1789581237) [ 745.024294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.080520] LNet: 33834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.108251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.173544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 745.214101] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 746.773406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[39-43]@tcp [ 746.815409] LNet: 33968:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 746.869187] LNetError: 33839:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 746.870586] LNetError: 33970:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.42@tcp has gone from up to down [ 746.888604] LNetError: 33970:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 755.338736] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:54:08 (1789581248) [ 756.830328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg438-server [ 758.497245] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:54:11 (1789581251) [ 758.701894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.723675] LNet: 34919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.734695] LNetError: 34919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.763407] LNet: Removed LNI 192.168.204.38@tcp [ 758.870879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.016690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.063938] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 760.027335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[39-44/2]@tcp [ 760.068739] LNet: 35036:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 760.080878] LNet: 35036:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 760.105126] LNetError: 34923:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 760.107798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[39-44/2]@tcp [ 760.114033] LNetError: 34923:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 760.114077] LNetError: 34923:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.39@tcp has gone from up to down [ 760.114081] LNetError: 34923:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 767.937802] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:54:20 (1789581260) [ 768.058646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.091859] LNet: 35630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.106782] LNetError: 35630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.139353] LNet: Removed LNI 192.168.204.38@tcp [ 768.170935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.270827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 768.375571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.464276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.578356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 768.713428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.768431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.818531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 768.903052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.953993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.013589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 769.100911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.204220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.294154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 769.387659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.414914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.524026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 769.631624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.671264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.730972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 769.797308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.834258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.890886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 769.970374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.005062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.080222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 770.191361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.260427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.371172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 770.487226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.537521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.641507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 770.724932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.828411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.010980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 778.580936] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:54:31 (1789581271) [ 781.025337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg438-server [ 782.871933] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:54:35 (1789581275) [ 785.478484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg438-server [ 787.582770] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:54:40 (1789581280) [ 787.827902] LNet: 37059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.835872] LNet: 37059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 788.555656] Key type .llcrypt unregistered [ 788.559118] Key type ._llcrypt unregistered [ 796.365466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 796.637780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 796.644875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 796.655132] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 799.637342] Key type ._llcrypt registered [ 799.640270] Key type .llcrypt registered [ 799.830785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.847871] alg: No test for adler32 (adler32-zlib) [ 800.748321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.813128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.850550] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 801.584391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 801.607050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 803.862413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 803.890472] LNet: Removed LNI 10.1.2.3@tcp [ 804.001813] LNet: 37779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.005925] LNetError: 37779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.019789] LNet: Removed LNI 192.168.204.38@tcp [ 804.628625] Key type .llcrypt unregistered [ 804.630126] Key type ._llcrypt unregistered [ 819.286616] Key type ._llcrypt registered [ 819.288397] Key type .llcrypt registered [ 820.918463] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:55:13 (1789581313) [ 821.374503] Key type .llcrypt unregistered [ 821.377735] Key type ._llcrypt unregistered [ 827.549316] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 827.762135] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.783916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 827.799408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.318757] Key type ._llcrypt registered [ 830.322606] Key type .llcrypt registered [ 830.533932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.550473] alg: No test for adler32 (adler32-zlib) [ 831.582512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.649932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.691799] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 832.568372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 832.596662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.627424] LNet: 39281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.639982] LNetError: 39281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 835.650842] LNet: Removed LNI 10.1.2.3@tcp [ 835.670558] LNet: Removed LNI 192.168.204.38@tcp [ 836.531457] Key type .llcrypt unregistered [ 836.534752] Key type ._llcrypt unregistered [ 852.404739] Key type ._llcrypt registered [ 852.407665] Key type .llcrypt registered [ 854.324797] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:55:46 (1789581346) [ 854.974366] Key type .llcrypt unregistered [ 854.975734] Key type ._llcrypt unregistered [ 861.549791] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.630267] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.636404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.644650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.211299] Key type ._llcrypt registered [ 864.213517] Key type .llcrypt registered [ 864.372137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.410858] alg: No test for adler32 (adler32-zlib) [ 865.347402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.379257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 865.423772] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 867.054422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 867.069191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.119438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 870.142557] LNet: Removed LNI 10.1.2.3@tcp [ 870.157843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 870.192068] LNet: Removed LNI 10.1.2.31@tcp [ 870.215231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 870.240973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.329813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 871.360930] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 873.355686] LNet: 41205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.364371] LNetError: 41205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 873.370731] LNet: Removed LNI 10.1.2.31@tcp [ 873.400433] LNet: Removed LNI 10.1.2.3@tcp [ 874.571875] Key type .llcrypt unregistered [ 874.576398] Key type ._llcrypt unregistered [ 890.741499] Key type ._llcrypt registered [ 890.743531] Key type .llcrypt registered [ 892.551933] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:56:24 (1789581384) [ 893.336822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.390296] alg: No test for adler32 (adler32-zlib) [ 894.412932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.531057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.575120] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 895.601585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.621830] LNet: 42166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.628261] LNetError: 42166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.643648] LNet: Removed LNI 192.168.204.38@tcp [ 895.676575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.767573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.38@tcp [ 895.820067] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 896.997470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.38@tcp [ 897.046720] LNet: Removed LNI 192.168.204.38@tcp [ 897.149328] LNet: 42301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.183817] Key type .llcrypt unregistered [ 898.188859] Key type ._llcrypt unregistered [ 916.593853] Key type ._llcrypt registered [ 916.597696] Key type .llcrypt registered [ 918.564887] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:56:50 (1789581410) [ 921.540051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg438-server [ 925.148443] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:56:56 (1789581416) [ 926.426464] Key type .llcrypt unregistered [ 926.439371] Key type ._llcrypt unregistered [ 934.864423] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 935.378939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 935.382421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 935.408994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.804727] Key type ._llcrypt registered [ 936.806856] Key type .llcrypt registered [ 937.017682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.028836] alg: No test for adler32 (adler32-zlib) [ 938.085976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.273374] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 939.418975] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 943.031938] LNet: 44148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.056695] LNetError: 44148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.075185] LNet: Removed LNI 192.168.204.38@tcp1 [ 943.106942] LNet: Removed LNI 10.1.2.3@tcp2 [ 944.497851] Key type .llcrypt unregistered [ 944.501906] Key type ._llcrypt unregistered [ 958.734222] Key type ._llcrypt registered [ 958.737207] Key type .llcrypt registered [ 960.465556] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:57:32 (1789581452) [ 960.947555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.974037] alg: No test for adler32 (adler32-zlib) [ 961.968869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.063133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.102352] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 962.899586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.923642] LNet: 45125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.934431] LNetError: 45125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.953102] LNet: Removed LNI 192.168.204.38@tcp [ 962.977480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.065405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 963.095243] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 970.874575] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:57:43 (1789581463) [ 970.996410] LNet: 45716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.007839] LNetError: 45716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.068820] LNet: Removed LNI 192.168.204.38@tcp [ 971.761634] Key type .llcrypt unregistered [ 971.765930] Key type ._llcrypt unregistered [ 979.123592] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.425372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 979.432980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 979.449064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 982.078522] Key type ._llcrypt registered [ 982.080984] Key type .llcrypt registered [ 982.379284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.412256] alg: No test for adler32 (adler32-zlib) [ 983.395527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.486122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 983.560647] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 984.771264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 984.810500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.430743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.500736] LNet: 46467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.514302] LNetError: 46467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 988.521733] LNet: Removed LNI 10.1.2.3@tcp [ 988.540683] LNet: Removed LNI 192.168.204.38@tcp [ 988.587295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.718744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 988.764902] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 988.769706] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.242790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.268862] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.277741] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.284916] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 989.291697] LNet: Removed LNI 10.1.2.3@tcp [ 989.309237] LNet: Removed LNI 192.168.204.38@tcp [ 989.337444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.429601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 989.472325] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 989.485754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.365760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 990.394556] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 991.982537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.044488] LNet: 46656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.053912] LNetError: 46656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.064799] LNetError: 46656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 992.094855] LNet: Removed LNI 10.1.2.3@tcp [ 992.105110] LNet: Removed LNI 192.168.204.38@tcp [ 992.119907] LNet: Removed LNI 192.168.204.38@tcp2 [ 992.188857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.256543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 992.294459] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 992.302696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.309436] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 1002.998663] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:58:15 (1789581495) [ 1003.139455] LNet: 47258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.159561] LNetError: 47258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.188594] LNetError: 47258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1003.201277] LNet: Removed LNI 10.1.2.3@tcp [ 1003.202847] LNet: Removed LNI 192.168.204.38@tcp [ 1003.243571] LNet: Removed LNI 192.168.204.38@tcp2 [ 1004.334825] Key type .llcrypt unregistered [ 1004.339423] Key type ._llcrypt unregistered [ 1010.786200] Key type ._llcrypt registered [ 1010.788254] Key type .llcrypt registered [ 1010.948284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.981644] alg: No test for adler32 (adler32-zlib) [ 1011.967311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.037810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1012.061610] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1013.306941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1013.339619] LNet: 47656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.345805] LNetError: 47656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.366743] LNet: Removed LNI 192.168.204.38@tcp [ 1013.407655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.499312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1013.530045] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1014.380307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1014.400618] LNet: 47687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.407098] LNetError: 47687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.420648] LNet: Removed LNI 192.168.204.38@tcp [ 1014.460619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.603116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1014.638809] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1015.277844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.321254] LNet: Removed LNI 192.168.204.38@tcp [ 1015.373034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.478489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1015.544105] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1016.267875] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1016.562462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1016.582094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1016.589215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1019.229955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.309765] LNet: 47798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.328091] LNet: 47798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1019.341387] LNetError: 47798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.348458] LNetError: 47798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1019.366204] LNet: Removed LNI 192.168.204.38@tcp [ 1019.411499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.517917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1019.560207] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1030.683434] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:58:43 (1789581523) [ 1030.895220] LNet: 48404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.899953] LNetError: 48404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.914240] LNet: Removed LNI 192.168.204.38@tcp [ 1031.734106] Key type .llcrypt unregistered [ 1031.735731] Key type ._llcrypt unregistered [ 1039.056605] Key type ._llcrypt registered [ 1039.063987] Key type .llcrypt registered [ 1039.433091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.475548] alg: No test for adler32 (adler32-zlib) [ 1040.383668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.432969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1040.493408] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1041.528360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.543987] LNet: 48801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.548976] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.570338] LNet: Removed LNI 192.168.204.38@tcp [ 1041.612414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.730801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1041.778427] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1049.708375] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:59:02 (1789581542) [ 1049.873807] LNet: 49396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.878965] LNetError: 49396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1049.913657] LNet: Removed LNI 192.168.204.38@tcp [ 1050.716418] Key type .llcrypt unregistered [ 1050.722259] Key type ._llcrypt unregistered [ 1062.461409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1062.656873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1062.659288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1062.661795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1064.530122] Key type ._llcrypt registered [ 1064.534674] Key type .llcrypt registered [ 1064.842598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1064.867461] alg: No test for adler32 (adler32-zlib) [ 1065.939907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.015157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.063642] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1067.209879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.255213] LNet: 49846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.268219] LNetError: 49846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.308406] LNet: Removed LNI 192.168.204.38@tcp [ 1067.371459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.464995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1067.510051] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1075.394044] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:59:27 (1789581567) [ 1075.529544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1075.578095] LNet: 50446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.594339] LNetError: 50446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.623213] LNet: Removed LNI 192.168.204.38@tcp [ 1075.642895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.794900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1083.401441] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:59:35 (1789581575) [ 1083.484820] LNet: 51036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.261624] Key type .llcrypt unregistered [ 1084.268611] Key type ._llcrypt unregistered [ 1090.406776] Key type ._llcrypt registered [ 1090.410323] Key type .llcrypt registered [ 1090.594432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.625377] alg: No test for adler32 (adler32-zlib) [ 1091.615619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.723304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1091.756959] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1093.029036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1093.062787] LNet: 51440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.075596] LNetError: 51440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1093.097251] LNet: Removed LNI 192.168.204.38@tcp [ 1093.136951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1093.236455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1093.262608] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1105.291774] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:59:56 (1789581596) [ 1105.455840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.525062] LNet: 52038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.547298] LNetError: 52038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.590888] LNet: Removed LNI 192.168.204.38@tcp [ 1105.662539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.772347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1115.676850] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:00:07 (1789581607) [ 1118.250280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1121.008919] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:00:13 (1789581613) [ 1122.806160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1124.980383] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:00:17 (1789581617) [ 1126.933351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1129.129232] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:00:21 (1789581621) [ 1130.939858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1133.070599] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:00:25 (1789581625) [ 1134.935892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1137.210518] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:00:29 (1789581629) [ 1137.482071] LNet: 54094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.167824] Key type .llcrypt unregistered [ 1138.172849] Key type ._llcrypt unregistered [ 1146.089690] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1146.392670] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1146.406902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1146.420687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1149.162557] Key type ._llcrypt registered [ 1149.165782] Key type .llcrypt registered [ 1149.417410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.433741] alg: No test for adler32 (adler32-zlib) [ 1150.397218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.551482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1150.594444] LNet: Added LNI 192.168.204.38@tcp [8/512/0/180] [ 1150.606615] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1150.610664] LNet: 54413:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1150.773690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1150.799785] LNet: 54432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.807742] LNetError: 54432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1150.815733] LNet: Removed LNI 10.1.2.3@tcp [ 1150.820383] LNetError: 54432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.833624] LNet: Removed LNI 192.168.204.38@tcp [ 1150.851051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.926833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1150.955949] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1151.009789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1151.026168] LNet: 54456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.031811] LNetError: 54456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.050589] LNet: Removed LNI 192.168.204.38@tcp [ 1151.072547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.152976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1151.181336] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1151.186139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.188854] LNet: 54464:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1151.332815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1151.362868] LNet: 54482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.368250] LNetError: 54482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.375382] LNet: Removed LNI 10.1.2.3@tcp [ 1151.377989] LNetError: 54482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.397197] LNet: Removed LNI 192.168.204.38@tcp [ 1151.461892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.596711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1151.633938] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1151.660500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.664370] LNet: 54490:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1151.948382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1151.969799] LNet: 54516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.974671] LNetError: 54516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.980859] LNet: Removed LNI 10.1.2.3@tcp [ 1151.985821] LNetError: 54516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.997514] LNet: Removed LNI 192.168.204.38@tcp [ 1152.061540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.168320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1152.208987] LNet: Added LNI 192.168.204.38@tcp [8/256/0/182] [ 1152.218305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1152.332429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1152.351411] LNet: 54545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.358580] LNetError: 54545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1152.364566] LNet: Removed LNI 10.1.2.3@tcp [ 1152.366801] LNetError: 54545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.384550] LNet: Removed LNI 192.168.204.38@tcp [ 1152.401831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.494731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1152.530251] LNet: Added LNI 192.168.204.38@tcp [8/256/0/182] [ 1152.542194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1152.546871] LNet: 54553:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1152.638369] LNet: Removed LNI 10.1.2.3@tcp [ 1152.655698] LNet: Removed LNI 192.168.204.38@tcp [ 1153.434843] Key type .llcrypt unregistered [ 1153.441912] Key type ._llcrypt unregistered [ 1167.610270] Key type ._llcrypt registered [ 1167.612675] Key type .llcrypt registered [ 1169.117638] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:01:01 (1789581661) [ 1170.773623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1172.568701] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:01:05 (1789581665) [ 1172.957108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.980735] alg: No test for adler32 (adler32-zlib) [ 1173.981812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1174.046530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1174.069846] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1174.916340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.38 [ 1174.954797] LNet: 55842:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1174.970519] LNet: Removed LNI 192.168.204.38@tcp [ 1181.656785] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:01:14 (1789581674) [ 1181.801878] LNet: 56413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.812360] LNetError: 56413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.856170] LNet: Removed LNI 192.168.204.38@tcp [ 1182.583516] Key type .llcrypt unregistered [ 1182.587724] Key type ._llcrypt unregistered [ 1189.631207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1190.012616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1193.014462] Key type ._llcrypt registered [ 1193.016215] Key type .llcrypt registered [ 1193.182674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1193.197252] alg: No test for adler32 (adler32-zlib) [ 1194.801584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1203.097564] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:01:35 (1789581695) [ 1203.317484] LNet: 57341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.323935] LNetError: 57341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1203.338704] LNet: Removed LNI 10.1.2.3@tcp [ 1204.151863] Key type .llcrypt unregistered [ 1204.156997] Key type ._llcrypt unregistered [ 1212.162818] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1212.686987] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1212.698991] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1212.717129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1215.830957] Key type ._llcrypt registered [ 1215.836933] Key type .llcrypt registered [ 1216.186897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1216.214129] alg: No test for adler32 (adler32-zlib) [ 1217.130519] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1226.185287] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:01:58 (1789581718) [ 1228.061810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1229.853670] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:02:02 (1789581722) [ 1230.125466] LNet: 58538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.133913] LNetError: 58538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1230.163784] LNet: Removed LNI 10.1.2.3@tcp [ 1231.078053] Key type .llcrypt unregistered [ 1231.082380] Key type ._llcrypt unregistered [ 1238.480472] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1238.835529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1238.841319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1238.845903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1241.806650] Key type ._llcrypt registered [ 1241.808894] Key type .llcrypt registered [ 1242.133338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.164262] alg: No test for adler32 (adler32-zlib) [ 1243.207035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.305954] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1253.275709] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:02:26 (1789581746) [ 1253.867421] LNet: 59776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.878617] LNetError: 59776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1253.893922] LNet: Removed LNI 10.1.2.3@tcp [ 1254.554396] Key type .llcrypt unregistered [ 1254.555985] Key type ._llcrypt unregistered [ 1261.738482] Key type ._llcrypt registered [ 1261.741628] Key type .llcrypt registered [ 1261.898994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1261.913287] alg: No test for adler32 (adler32-zlib) [ 1262.838803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1262.857428] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1271.097599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1271.991878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_interrupt -r 1 [ 1272.036328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_interrupt -r 1 [ 1272.077550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1272.103508] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1272.349550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.389045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.416286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.441384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.474787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1272.498567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.528258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.559215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.593453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.613159] LNet: 60216:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.631348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.661958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.711428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1272.758983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1272.791616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1272.836776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1272.877604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1273.509295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_dropped -r 1 [ 1273.586441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_dropped -r 1 [ 1273.646425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1273.690629] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1273.998371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.066452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.131709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.147625] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1274.194357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.271511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1274.328950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.385720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.442538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.486182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.519037] LNet: 60327:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.559555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.594979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.641986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.682984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.753863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.794710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.831237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1275.361968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_aborted -r 1 [ 1275.408875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_aborted -r 1 [ 1275.448659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1275.468424] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1275.473788] LNet: Skipped 1 previous similar message [ 1275.672396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.702982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.731953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.761705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.791657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1275.824302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.848446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.878473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.904706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.918945] LNet: 60439:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.934250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.964633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.998714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.036167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.068898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.100481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.131422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1276.623430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_no_route -r 1 [ 1276.664396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_no_route -r 1 [ 1276.699844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1276.890175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.916953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.945716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.995318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.032528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1277.066968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.097538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.125114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.154071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.190919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.228569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.274282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.349680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.388582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.418628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.457066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.063689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_timeout -r 1 [ 1278.106279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_timeout -r 1 [ 1278.142528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1278.176242] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1278.184316] LNet: Skipped 1 previous similar message [ 1278.475516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.499910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.522666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.557464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.593369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1278.621440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.658201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.689659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.726670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.751469] LNet: 60661:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.757873] LNet: 60661:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1278.772865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.810237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.855747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.880841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.933496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.961691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.996980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1279.460520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_error -r 1 [ 1279.507716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_error -r 1 [ 1279.554524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1279.872744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.905433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.940421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.978662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.027933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.058523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.095950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.130860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.181076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.230360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.293670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.353395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.400709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.448565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.494845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.554468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.818590] LNet: 60814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.824281] LNetError: 60814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.841475] LNet: Removed LNI 192.168.204.38@tcp [ 1281.185356] Key type .llcrypt unregistered [ 1281.189189] Key type ._llcrypt unregistered [ 1300.161179] Key type ._llcrypt registered [ 1300.163660] Key type .llcrypt registered [ 1302.346820] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:03:14 (1789581794) [ 1303.578182] Key type .llcrypt unregistered [ 1303.582532] Key type ._llcrypt unregistered [ 1312.274874] Key type ._llcrypt registered [ 1312.284237] Key type .llcrypt registered [ 1312.534897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.574757] alg: No test for adler32 (adler32-zlib) [ 1313.560129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1313.598274] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1322.866236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1336.270810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.314052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.348670] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1338.023452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_interrupt -r 1 [ 1338.051919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_interrupt -r 1 [ 1338.095710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_interrupt -r 1 [ 1338.189343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_interrupt -r 1 [ 1338.228576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_interrupt -r 1 [ 1338.269199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_interrupt -r 1 [ 1338.322618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_interrupt -r 1 [ 1338.396041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_interrupt -r 1 [ 1338.466364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1338.501688] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1338.513546] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1338.808268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.844866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.876229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.932229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.966246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1339.035340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.058536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.100925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.138890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.158176] LNet: 62294:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.201885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.237520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.293164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.333994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.387430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.452691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.497257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.975166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_dropped -r 1 [ 1340.042783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_dropped -r 1 [ 1340.093791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_dropped -r 1 [ 1340.137214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_dropped -r 1 [ 1340.171268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_dropped -r 1 [ 1340.218381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_dropped -r 1 [ 1340.290766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_dropped -r 1 [ 1340.376748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_dropped -r 1 [ 1340.452539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1340.493726] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1340.499657] LNet: Skipped 2 previous similar messages [ 1340.774357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.803558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.833398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.854722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.880578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1340.927767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.988176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.022314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.048153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.096098] LNet: 62417:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.142725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.212924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.254120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.282179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.310420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.343210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.435884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.104520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_aborted -r 1 [ 1342.137299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_aborted -r 1 [ 1342.184881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_aborted -r 1 [ 1342.220688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_aborted -r 1 [ 1342.280886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_aborted -r 1 [ 1342.327648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_aborted -r 1 [ 1342.379457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_aborted -r 1 [ 1342.417813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_aborted -r 1 [ 1342.472158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1342.510047] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1342.513967] LNet: Skipped 2 previous similar messages [ 1342.746486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.790760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.835670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.884460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.953878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1343.004960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.077367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.141356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.202576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.218874] LNet: 62541:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.237435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.274910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.347028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.383224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.445790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.481873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.531398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1344.233551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_no_route -r 1 [ 1344.266447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_no_route -r 1 [ 1344.325975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_no_route -r 1 [ 1344.371817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_no_route -r 1 [ 1344.406783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_no_route -r 1 [ 1344.437504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_no_route -r 1 [ 1344.467169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_no_route -r 1 [ 1344.533299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_no_route -r 1 [ 1344.587696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1344.609898] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1344.615573] LNet: Skipped 2 previous similar messages [ 1344.978947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1345.027251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1345.120188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1345.184872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1345.221670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1345.277414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1345.314154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1345.355229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1345.429994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1345.457869] LNet: 62664:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.512400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1345.604673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.666058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.700387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.723833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.745663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1345.766483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1346.322812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_timeout -r 1 [ 1346.352318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_timeout -r 1 [ 1346.383836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_timeout -r 1 [ 1346.432122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_timeout -r 1 [ 1346.470243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_timeout -r 1 [ 1346.522491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_timeout -r 1 [ 1346.585826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_timeout -r 1 [ 1346.668775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_timeout -r 1 [ 1346.738960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1347.016048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.039352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1347.059560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.080826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.107504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1347.135583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.166531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.195053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.225266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.248126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.275185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.325818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.393888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.439874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.498915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1347.544328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1348.160653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_error -r 1 [ 1348.218570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_error -r 1 [ 1348.270933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_error -r 1 [ 1348.324655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_error -r 1 [ 1348.386322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_error -r 1 [ 1348.414566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_error -r 1 [ 1348.464797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_error -r 1 [ 1348.510290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_error -r 1 [ 1348.547127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1348.754693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.799918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.837358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.861141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.891240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1348.927270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.990372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.033140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.088551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.145377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.174987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.205640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.241867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.270203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.324555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.366194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1354.960193] LNet: 62986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.977261] LNetError: 62986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.000176] LNet: Removed LNI 192.168.204.38@tcp [ 1356.002042] LNetError: 62986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.018479] LNet: Removed LNI 192.168.204.38@tcp1 [ 1356.668888] Key type .llcrypt unregistered [ 1356.679469] Key type ._llcrypt unregistered [ 1376.068690] Key type ._llcrypt registered [ 1376.074718] Key type .llcrypt registered [ 1377.723349] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:04:30 (1789581870) [ 1378.655391] Key type .llcrypt unregistered [ 1378.662411] Key type ._llcrypt unregistered [ 1384.947458] Key type ._llcrypt registered [ 1384.950173] Key type .llcrypt registered [ 1385.078379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.096499] alg: No test for adler32 (adler32-zlib) [ 1385.974523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1386.006792] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1393.823750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1394.478395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_dropped -r 1 [ 1394.558229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_dropped -r 1 [ 1394.599765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1394.628657] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1394.872465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1394.908507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1394.950353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1394.972423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1395.011862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1395.056770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1395.094524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1395.159242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1395.190882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1395.219976] LNet: 64246:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1395.261922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1395.337898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1395.417988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1395.503174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1395.575980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1395.633103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1395.677558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1396.359102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_error -r 1 [ 1396.420979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_error -r 1 [ 1396.493712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1396.530381] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1396.835268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1396.905600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1396.950268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1396.995893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1397.039366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1397.112123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1397.154630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1397.202329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1397.234339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1397.249629] LNet: 64358:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1397.267784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1397.295371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1397.325384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1397.351944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1397.381221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1397.409815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1397.464175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1398.110616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_timeout -r 1 [ 1398.152800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_timeout -r 1 [ 1398.206324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1398.243583] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1398.538794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1398.575827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1398.607824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1398.637691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1398.662634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1398.692750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1398.736540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1398.789541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1398.841076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1398.868115] LNet: 64469:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1398.901796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1398.981678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1399.041780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.123863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1399.214238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1399.269086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.362339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1399.746548] LNet: 64510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.750478] LNetError: 64510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.776744] LNet: Removed LNI 192.168.204.38@tcp [ 1400.515712] Key type .llcrypt unregistered [ 1400.525992] Key type ._llcrypt unregistered [ 1419.262856] Key type ._llcrypt registered [ 1419.265661] Key type .llcrypt registered [ 1420.911911] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:05:13 (1789581913) [ 1422.167277] Key type .llcrypt unregistered [ 1422.168889] Key type ._llcrypt unregistered [ 1430.388943] Key type ._llcrypt registered [ 1430.400794] Key type .llcrypt registered [ 1430.682234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.704416] alg: No test for adler32 (adler32-zlib) [ 1431.706071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1431.751364] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1440.774833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1450.694419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.752610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1450.780383] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1452.444098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_dropped -r 1 [ 1452.502911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e remote_dropped -r 1 [ 1452.563149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_dropped -r 1 [ 1452.622123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e remote_dropped -r 1 [ 1452.656948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e remote_dropped -r 1 [ 1452.712112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e remote_dropped -r 1 [ 1452.764566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e remote_dropped -r 1 [ 1452.847960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e remote_dropped -r 1 [ 1452.902065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1452.935128] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1452.952137] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.138@tcp [ 1453.155216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1453.191645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1453.227639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1453.266789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1453.303727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1453.360678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1453.398909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1453.482921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1453.525607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1453.552692] LNet: 65990:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1453.585806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1453.626423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.668627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.711716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.744992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.778943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.812315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.850166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1454.343171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_error -r 1 [ 1454.381423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e remote_error -r 1 [ 1454.440283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_error -r 1 [ 1454.485489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e remote_error -r 1 [ 1454.534297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e remote_error -r 1 [ 1454.564842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e remote_error -r 1 [ 1454.597213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e remote_error -r 1 [ 1454.628700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e remote_error -r 1 [ 1454.666482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1454.706525] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1454.718767] LNet: Skipped 2 previous similar messages [ 1454.963677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1454.988782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1455.019859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.050294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.086301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1455.116467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.149992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.180963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.213964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.232940] LNet: 66115:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.259210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1455.303086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1455.361882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.405893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1455.433611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1455.464825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1455.499812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1455.546141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1456.036620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_timeout -r 1 [ 1456.076971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e remote_timeout -r 1 [ 1456.122200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_timeout -r 1 [ 1456.157424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e remote_timeout -r 1 [ 1456.201879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e remote_timeout -r 1 [ 1456.242705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e remote_timeout -r 1 [ 1456.282986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e remote_timeout -r 1 [ 1456.327991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e remote_timeout -r 1 [ 1456.371690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.138@tcp [ 1456.395051] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1456.401926] LNet: Skipped 1 previous similar message [ 1456.610866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1456.640262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1456.665841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1456.688851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1456.713877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1456.742502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1456.770945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1456.816683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1456.860488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1456.891147] LNet: 66240:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1456.922976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1456.961505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1457.005976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1457.043500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1457.089290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1457.132706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1457.176676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.232307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1462.206867] LNet: 66318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.221373] LNetError: 66318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1462.235286] LNet: Removed LNI 192.168.204.38@tcp [ 1462.261980] LNet: Removed LNI 192.168.204.38@tcp1 [ 1462.663499] Key type .llcrypt unregistered [ 1462.665914] Key type ._llcrypt unregistered [ 1478.966286] Key type ._llcrypt registered [ 1478.968546] Key type .llcrypt registered [ 1480.226575] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:06:13 (1789581973) [ 1482.142733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1484.105509] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:06:16 (1789581976) [ 1484.947457] Key type .llcrypt unregistered [ 1484.951092] Key type ._llcrypt unregistered [ 1491.881219] Key type ._llcrypt registered [ 1491.887867] Key type .llcrypt registered [ 1492.136925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1492.154262] alg: No test for adler32 (adler32-zlib) [ 1493.229726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1493.304428] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1501.077930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1501.829723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e network_timeout -r 1 [ 1501.903679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e network_timeout -r 1 [ 1501.973251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1501.995913] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1502.217324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1502.244558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1502.274610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1502.298644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1502.335375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1502.363316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1502.394564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1502.424802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1502.467332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1502.489264] LNet: 67875:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1502.513818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1502.549141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1502.564529] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1502.606927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1502.641408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1502.675289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1502.710342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1503.009224] LNet: 67915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.013594] LNetError: 67915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.039120] LNet: Removed LNI 192.168.204.38@tcp [ 1504.581728] Key type .llcrypt unregistered [ 1504.585349] Key type ._llcrypt unregistered [ 1523.846269] Key type ._llcrypt registered [ 1523.859739] Key type .llcrypt registered [ 1524.215671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1524.233894] alg: No test for adler32 (adler32-zlib) [ 1525.102502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1525.122112] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1535.659118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1547.094649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.173919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.224621] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1549.538884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e network_timeout -r 1 [ 1549.580092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e network_timeout -r 1 [ 1549.619312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e network_timeout -r 1 [ 1549.680667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e network_timeout -r 1 [ 1549.729635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e network_timeout -r 1 [ 1549.779772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e network_timeout -r 1 [ 1549.858318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e network_timeout -r 1 [ 1549.938332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e network_timeout -r 1 [ 1549.994286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1550.020666] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1550.427568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1550.503386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1550.572058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1550.648835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1550.676544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1550.711156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1550.747889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1550.760634] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1550.764302] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.138@tcp [ 1550.795123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1550.827901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1550.844126] LNet: 68797:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1550.863639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1550.901407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1550.954737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1551.014890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1551.056374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1551.123824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1555.898349] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1556.290604] LNet: 68871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.299217] LNetError: 68871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.344277] LNet: Removed LNI 192.168.204.38@tcp [ 1557.352789] LNetError: 68871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.385760] LNet: Removed LNI 192.168.204.38@tcp1 [ 1557.856962] Key type .llcrypt unregistered [ 1557.859353] Key type ._llcrypt unregistered [ 1576.994935] Key type ._llcrypt registered [ 1576.997520] Key type .llcrypt registered [ 1578.448682] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:07:51 (1789582071) [ 1578.827441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1578.842426] alg: No test for adler32 (adler32-zlib) [ 1579.825294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.878789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.901808] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1580.633444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1580.689802] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1581.672861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 1581.758350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1581.881660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1581.927454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1581.998625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1582.039736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1582.080988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.38@tcp [ 1582.112333] LNetError: 69979:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.38@tcp to recovery [ 1582.945431] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1582.946086] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1584.992506] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1585.785730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1588.067289] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1588.219945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1588.272030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1588.347416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.371286] LNet: 70076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.376929] LNetError: 70076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.382670] LNet: Removed LNI 192.168.204.38@tcp [ 1588.396623] LNet: Removed LNI 192.168.204.38@tcp1 [ 1588.417770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.464793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.481254] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1589.227413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.247774] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1589.993628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 1590.021466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1590.071597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1590.112179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1590.153589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1590.201849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1590.260281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1590.316995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.38@tcp [ 1590.363069] LNetError: 70348:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.38@tcp to recovery [ 1590.497687] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1592.840523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1596.640522] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1596.665160] LNet: Skipped 1 previous similar message [ 1601.871867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1601.915866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1601.955466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1609.004903] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:08:21 (1789582101) [ 1609.113720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.152876] LNet: 71066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.159900] LNetError: 71066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.164118] LNetError: 71066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1609.170797] LNet: Removed LNI 192.168.204.38@tcp [ 1609.188673] LNet: Removed LNI 192.168.204.38@tcp1 [ 1609.232145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.298695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.322606] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1610.331707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.362064] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1611.165726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 1611.241804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1611.287628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1611.319757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1611.356889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1611.384045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1611.415650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.38@tcp --health 0 [ 1611.563842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1612.384402] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.38@tcp [ 1612.385700] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1612.408216] LNet: Skipped 1 previous similar message [ 1627.166264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1627.395665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1627.421249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.441391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.38@tcp --health 500 [ 1627.515864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1634.190098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1634.266365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.303478] LNet: 71566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.318260] LNetError: 71566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.333198] LNetError: 71566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1634.345658] LNet: Removed LNI 192.168.204.38@tcp [ 1634.362393] LNet: Removed LNI 192.168.204.38@tcp1 [ 1634.389419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.467873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.498388] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1635.151415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1635.172578] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1635.933799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 1635.958915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.014200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1636.064843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1636.098235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1636.168457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1636.215246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1636.257836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.38@tcp --health 0 [ 1637.537453] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1637.546627] LNet: Skipped 3 previous similar messages [ 1638.669982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.024508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1649.066164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1649.108471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1656.936965] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:09:09 (1789582149) [ 1657.492642] LNet: 72633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.506377] LNetError: 72633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.517601] LNetError: 72633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1657.528493] LNet: Removed LNI 192.168.204.38@tcp [ 1657.538542] LNet: Removed LNI 192.168.204.38@tcp1 [ 1658.312759] Key type .llcrypt unregistered [ 1658.315251] Key type ._llcrypt unregistered [ 1665.623846] Key type ._llcrypt registered [ 1665.629891] Key type .llcrypt registered [ 1665.834591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1665.850106] alg: No test for adler32 (adler32-zlib) [ 1666.804658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1666.832811] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1676.353752] Lustre: DEBUG MARKER: Initial discovery [ 1676.435138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.138@tcp [ 1679.117336] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1679.182211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1679.234695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.138@tcp [ 1679.265334] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1679.293331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1679.339634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.138@tcp -r 1 -m GET -l 3 [ 1679.402634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.138@tcp -r 1 -m GET -e local_error [ 1679.446847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.138@tcp -r 1 -m PUT -l 6 [ 1680.167019] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1681.053339] Lustre: DEBUG MARKER: Force oleg438-client.virtnet to discover 192.168.204.138@tcp (in background) [ 1681.178753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.138@tcp [ 1682.733588] Lustre: DEBUG MARKER: Force oleg438-server to discover 192.168.204.38@tcp [ 1684.258318] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1685.050314] Lustre: DEBUG MARKER: Wait for 73400 [ 1686.643658] Lustre: DEBUG MARKER: Finished wait on 73400 [ 1686.778061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1686.815082] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1686.831303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1687.144783] LNet: 73856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.179135] LNetError: 73856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.218122] LNet: Removed LNI 192.168.204.38@tcp [ 1687.636640] Key type .llcrypt unregistered [ 1687.640776] Key type ._llcrypt unregistered [ 1702.927749] Key type ._llcrypt registered [ 1702.934397] Key type .llcrypt registered [ 1704.537197] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:09:57 (1789582197) [ 1705.280394] Key type .llcrypt unregistered [ 1705.286464] Key type ._llcrypt unregistered [ 1712.840390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1712.963205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1712.966143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1712.973513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1715.634512] Key type ._llcrypt registered [ 1715.638655] Key type .llcrypt registered [ 1715.816637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1715.845535] alg: No test for adler32 (adler32-zlib) [ 1716.882609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.987439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.033410] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1718.014226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1718.052836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1727.796125] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:10:20 (1789582220) [ 1727.909087] LNet: 75938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.915754] LNetError: 75938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1727.927233] LNet: Removed LNI 10.1.2.3@tcp [ 1727.957827] LNet: Removed LNI 192.168.204.38@tcp [ 1728.534644] Key type .llcrypt unregistered [ 1728.539728] Key type ._llcrypt unregistered [ 1735.182991] Key type ._llcrypt registered [ 1735.186651] Key type .llcrypt registered [ 1735.381122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1735.396795] alg: No test for adler32 (adler32-zlib) [ 1736.369182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.456324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1736.488961] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1737.524853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1737.561156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1738.801400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.38@tcp [ 1739.877869] LNetError: Unexpected error -22 connecting to 192.168.204.38@tcp at host 192.168.204.38:988 [ 1746.135118] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:10:38 (1789582238) [ 1746.284984] LNet: 77080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.301711] LNetError: 77080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1746.314445] LNet: Removed LNI 10.1.2.3@tcp [ 1746.328857] LNet: Removed LNI 192.168.204.38@tcp [ 1747.018557] Key type .llcrypt unregistered [ 1747.024607] Key type ._llcrypt unregistered [ 1754.058337] Key type ._llcrypt registered [ 1754.061696] Key type .llcrypt registered [ 1754.321318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1754.371251] alg: No test for adler32 (adler32-zlib) [ 1755.435634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.527655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1755.566331] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1756.583358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1756.610871] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 1757.427452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.38@tcp1 --nid 192.168.204.38@tcp2 [ 1761.952512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp1 192.168.204.38@tcp2 [ 1761.971890] LNetError: 77921:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.38@tcp1 to 192.168.204.38@tcp2 - Route table may be misconfigured [ 1761.987702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp2 192.168.204.38@tcp1 [ 1768.345139] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:11:01 (1789582261) [ 1768.514265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.554651] LNet: 78499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.564887] LNetError: 78499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.571256] LNet: Removed LNI 192.168.204.38@tcp1 [ 1769.645156] LNet: Removed LNI 192.168.204.38@tcp2 [ 1769.677810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.807907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.856482] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1770.833810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1770.849898] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1771.695715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 1771.741441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.38@tcp [ 1771.792828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp1 192.168.204.38@tcp1 [ 1771.847970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.38@tcp -d 192.168.204.38@tcp -e network_timeout [ 1771.894906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -e network_timeout [ 1771.937891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -e network_timeout [ 1771.985766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -e network_timeout [ 1772.042460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1772.089534] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1772.131142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1772.264943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1772.373650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1772.832122] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1779.222930] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:11:12 (1789582272) [ 1779.370799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.396493] LNet: 79366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.404705] LNetError: 79366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.411119] LNetError: 79366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1779.416984] LNet: Removed LNI 192.168.204.38@tcp [ 1780.460435] LNet: Removed LNI 192.168.204.38@tcp1 [ 1780.508936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.603777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1781.038566] LNet: 79409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.454936] Key type .llcrypt unregistered [ 1781.460893] Key type ._llcrypt unregistered [ 1793.670919] Key type ._llcrypt registered [ 1793.675435] Key type .llcrypt registered [ 1794.868578] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:11:27 (1789582287) [ 1795.357995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.372253] alg: No test for adler32 (adler32-zlib) [ 1796.320369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.847714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1799.672099] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:11:32 (1789582292) [ 1799.759413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.802394] LNet: 80510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.832225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.925430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.957880] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1800.672540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1800.689029] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1801.564621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1801.596372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp1 [ 1801.629153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp1 [ 1808.548264] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:11:41 (1789582301) [ 1810.407383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg438-server [ 1812.250913] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:11:45 (1789582305) [ 1814.410784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg438-server [ 1816.156661] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:11:49 (1789582309) [ 1818.120919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg438-server [ 1819.701487] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:11:52 (1789582312) [ 1821.694580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg438-server [ 1823.402172] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:11:56 (1789582316) [ 1825.339457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg438-server [ 1827.362976] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:12:00 (1789582320) [ 1829.480906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg438-server [ 1831.350812] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:12:04 (1789582324) [ 1833.559147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg438-server [ 1835.358583] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:12:08 (1789582328) [ 1837.265297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg438-server [ 1839.158594] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:12:12 (1789582332) [ 1841.102761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg438-server [ 1843.034647] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:12:15 (1789582335) [ 1843.123124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.159489] LNet: 84449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.173619] LNetError: 84449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.193211] LNet: Removed LNI 192.168.204.38@tcp [ 1844.201759] LNetError: 84449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.219297] LNet: Removed LNI 192.168.204.38@tcp1 [ 1844.253258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.304864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.335368] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1844.967451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1845.034595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1845.175605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.194337] LNet: 84605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1845.204039] LNetError: 84605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.257524] LNet: Removed LNI 192.168.204.38@tcp [ 1846.291786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.428962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1846.454430] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1847.277783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1847.495572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.527731] LNet: 84750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.539196] LNetError: 84750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.618958] LNet: Removed LNI 192.168.204.38@tcp [ 1848.644842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.722422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.764647] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1849.513240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1849.586552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1849.748563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.790687] LNet: Removed LNI 192.168.204.38@tcp [ 1850.826825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.904747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1850.933119] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1851.712990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1851.989050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.044510] LNet: 85043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.049390] LNet: 85043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1852.066072] LNetError: 85043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.073385] LNetError: 85043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1852.100248] LNet: Removed LNI 192.168.204.38@tcp [ 1852.137541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.224520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.242962] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1853.194185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1853.379775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1853.647154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.696052] LNet: Removed LNI 192.168.204.38@tcp [ 1854.723701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.809351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1854.845546] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1855.840512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1856.122943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.190511] LNet: Removed LNI 192.168.204.38@tcp [ 1857.272834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.341850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1857.365986] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1858.060047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1858.203513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1858.443875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.495968] LNet: Removed LNI 192.168.204.38@tcp [ 1859.544917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.615926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1859.660247] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1860.588687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1860.844647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.869944] LNet: 85629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.878899] LNet: 85629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1860.883730] LNetError: 85629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.891294] LNetError: 85629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1861.928347] LNet: Removed LNI 192.168.204.38@tcp [ 1861.982658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1862.092931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1862.156455] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1863.059986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1863.150361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1863.442592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.485855] LNet: Removed LNI 192.168.204.38@tcp [ 1864.509819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.659725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1864.701166] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1865.796896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1867.303750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.390231] LNet: Removed LNI 192.168.204.38@tcp [ 1868.420924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.503841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.555264] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1869.209065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1869.294925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1869.577765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.634077] LNet: Removed LNI 192.168.204.38@tcp [ 1870.685221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.800751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1870.850126] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1871.477311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 1871.705545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.760073] LNet: Removed LNI 192.168.204.38@tcp [ 1872.783461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.848667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.871240] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1873.496937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1873.563828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.606818] LNet: Removed LNI 192.168.204.38@tcp [ 1873.642767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.709466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.744985] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1874.655974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1874.761963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1874.809913] LNet: Removed LNI 192.168.204.38@tcp [ 1874.846586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1874.924418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1875.019148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.091631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.170567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1875.299867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.349980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.444991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1875.605262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.669477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1875.759651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1875.818982] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1876.553550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.584798] LNet: Removed LNI 192.168.204.38@tcp [ 1876.604169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.688385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1884.095981] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:12:56 (1789582376) [ 1884.236875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.272529] LNet: 87249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.283563] LNet: 87249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1884.342183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.420673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1884.442527] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/180] [ 1886.107710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.129786] LNetError: 87392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.137330] LNetError: 87392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1886.172046] LNet: Removed LNI 192.168.204.38@tcp231 [ 1886.220957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.346879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1886.399706] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/180] [ 1886.414727] LNet: 87400:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.173084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1887.203301] LNet: Removed LNI 192.168.204.38@tcp231 [ 1887.218267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1887.235561] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/180] [ 1888.519248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.538796] LNet: Removed LNI 192.168.204.38@tcp231 [ 1888.552754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.601932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1888.631340] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/0] [ 1888.635552] LNet: 87552:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1888.870608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1888.904985] LNet: Removed LNI 192.168.204.38@tcp231 [ 1888.925620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1888.964370] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/0] [ 1890.194159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1890.244888] LNet: Removed LNI 192.168.204.38@tcp231 [ 1890.277913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.342917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1890.366460] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/60] [ 1890.374267] LNet: 87704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.823659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1890.862532] LNet: Removed LNI 192.168.204.38@tcp231 [ 1890.876724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1890.919078] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/60] [ 1892.193116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.237135] LNet: Removed LNI 192.168.204.38@tcp231 [ 1892.284875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.373530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1892.427871] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/180] [ 1892.435217] LNet: 87855:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1892.936703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1892.962679] LNet: Removed LNI 192.168.204.38@tcp231 [ 1892.978379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1893.004760] LNet: Added LNI 192.168.204.38@tcp231 [8/256/0/180] [ 1901.854583] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:13:14 (1789582394) [ 1901.925028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.958891] LNet: Removed LNI 192.168.204.38@tcp231 [ 1901.977661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.076578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1902.095984] LNet: Added LNI 192.168.204.38@tcp232 [8/256/0/180] [ 1903.186877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1903.249896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1903.294279] LNet: Removed LNI 192.168.204.38@tcp232 [ 1903.322716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1903.381124] LNet: Added LNI 192.168.204.38@tcp232 [8/256/0/180] [ 1904.658312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1904.749334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1904.804607] LNet: Removed LNI 192.168.204.38@tcp232 [ 1904.846798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1904.887959] LNet: Added LNI 192.168.204.38@tcp232 [8/256/0/180] [ 1905.973971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1906.049427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1906.082827] LNet: Removed LNI 192.168.204.38@tcp232 [ 1906.119658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1906.222449] LNet: Added LNI 192.168.204.38@tcp232 [8/256/0/180] [ 1907.731722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1907.838994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1907.895801] LNet: Removed LNI 192.168.204.38@tcp232 [ 1907.922974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1916.551252] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:13:29 (1789582409) [ 1916.963217] LNet: 89713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.967156] LNet: 89713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1917.424512] Key type .llcrypt unregistered [ 1917.427168] Key type ._llcrypt unregistered [ 1925.284558] Key type ._llcrypt registered [ 1925.289933] Key type .llcrypt registered [ 1925.633575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1925.678352] alg: No test for adler32 (adler32-zlib) [ 1926.657933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1926.690790] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1935.494555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 1946.398729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.438176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1946.454824] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1947.722802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_interrupt -r 2 [ 1947.769960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_interrupt -r 2 [ 1947.826555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_interrupt -r 2 [ 1947.871836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_interrupt -r 2 [ 1947.941560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_interrupt -r 2 [ 1948.005430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_interrupt -r 2 [ 1948.067348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_interrupt -r 2 [ 1948.137692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_interrupt -r 2 [ 1948.187499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1948.208208] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1948.213717] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1948.480660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1948.504793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1948.548260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1948.595360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1948.650438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1948.703308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1948.749614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1948.779916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1948.812496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1948.841983] LNet: 90377:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1948.871383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1948.916715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1948.968933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1949.009616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1949.052210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1949.103070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1949.767436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_dropped -r 2 [ 1949.820523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_dropped -r 2 [ 1949.878448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_dropped -r 2 [ 1949.965349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_dropped -r 2 [ 1950.036241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_dropped -r 2 [ 1950.086320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_dropped -r 2 [ 1950.138273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_dropped -r 2 [ 1950.195220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_dropped -r 2 [ 1950.276616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1950.317531] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1950.645686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1950.689338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1950.733213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1950.773373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1950.827273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1950.906767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1950.939606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1950.975841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1951.036239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1951.070675] LNet: 90500:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.101615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1951.144069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1951.198677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1951.263348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1951.297573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1951.331313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1951.940250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_aborted -r 2 [ 1951.975913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_aborted -r 2 [ 1952.023404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_aborted -r 2 [ 1952.062423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_aborted -r 2 [ 1952.099366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_aborted -r 2 [ 1952.142602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_aborted -r 2 [ 1952.185699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_aborted -r 2 [ 1952.246368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_aborted -r 2 [ 1952.305974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1952.349544] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1952.618577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1952.691161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1952.734952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1952.804121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1952.881379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1952.947237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1952.991824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1953.067513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1953.126788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1953.161094] LNet: 90622:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1953.191975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1953.235529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1953.300300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1953.339655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1953.374922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1953.407988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1954.065367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_no_route -r 2 [ 1954.105130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_no_route -r 2 [ 1954.134799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_no_route -r 2 [ 1954.184053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_no_route -r 2 [ 1954.227212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_no_route -r 2 [ 1954.283565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_no_route -r 2 [ 1954.317216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_no_route -r 2 [ 1954.350641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_no_route -r 2 [ 1954.386990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1954.410562] LNet: There was an unexpected network error while writing to 192.168.204.138: rc = -22 [ 1954.649353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1954.712603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1954.756644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1954.813292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1954.845588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1954.902069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1954.939485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1954.989478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1955.040511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1955.070990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1955.096869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1955.118466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1955.149080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1955.196781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1955.227452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1955.764592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e local_timeout -r 2 [ 1955.806748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e local_timeout -r 2 [ 1955.837575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e local_timeout -r 2 [ 1955.882933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e local_timeout -r 2 [ 1955.945245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e local_timeout -r 2 [ 1955.988657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e local_timeout -r 2 [ 1956.052898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e local_timeout -r 2 [ 1956.107901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e local_timeout -r 2 [ 1956.167206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1956.527332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1956.573566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1956.612021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1956.641455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1956.675270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1956.734662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1956.801280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1956.854498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1956.905047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1956.963494] LNet: 90867:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1956.974977] LNet: 90867:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1957.009839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1957.050497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1957.086435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1957.142433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1957.167169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1957.200497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1957.660617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp -m GET -e remote_dropped -r 2 [ 1957.730498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp -d 192.168.204.138@tcp1 -m GET -e remote_dropped -r 2 [ 1957.818356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp -m GET -e remote_dropped -r 2 [ 1957.906344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp -d 192.168.204.38@tcp1 -m GET -e remote_dropped -r 2 [ 1957.932569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp -m GET -e remote_dropped -r 2 [ 1957.991564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.138@tcp1 -m GET -e remote_dropped -r 2 [ 1958.039895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp -m GET -e remote_dropped -r 2 [ 1958.085147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.38@tcp1 -d 192.168.204.38@tcp1 -m GET -e remote_dropped -r 2 [ 1958.116260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.38@tcp 192.168.204.138@tcp [ 1958.136376] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.138@tcp [ 1958.304253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1958.345754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1958.417680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1958.460043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1958.487890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1958.514388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1958.563187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1958.593525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1958.621936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1958.665683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1958.702070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1958.741878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1958.793418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1958.836333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1958.864663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1964.182244] LNet: 91064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.194130] LNetError: 91064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.210083] LNet: Removed LNI 192.168.204.38@tcp [ 1964.228636] LNet: Removed LNI 192.168.204.38@tcp1 [ 1964.668846] Key type .llcrypt unregistered [ 1964.671248] Key type ._llcrypt unregistered [ 1982.257338] Key type ._llcrypt registered [ 1982.262074] Key type .llcrypt registered [ 1983.681724] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:14:36 (1789582476) [ 1984.162753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.181501] alg: No test for adler32 (adler32-zlib) [ 1985.115028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.168267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.193880] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1986.347590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1986.388666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.38@tcp [ 1986.418480] LNetError: 92051:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.38@tcp to recovery [ 1986.551024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.38@tcp [ 1986.698784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1992.557332] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:14:45 (1789582485) [ 1992.731730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.785036] LNet: 92648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.794861] LNetError: 92648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.822314] LNet: Removed LNI 192.168.204.38@tcp [ 1992.865373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.935580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1992.970571] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 1993.710199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1993.727715] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 1994.802517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.38@tcp [ 1994.823620] LNetError: 92915:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.38@tcp to recovery [ 1994.976362] LNet: 1 local NIs in recovery (showing 1): 192.168.204.38@tcp [ 1994.982488] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1996.002494] LNet: There was an unexpected network error while writing to 192.168.204.38: rc = -22 [ 1996.106814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.38@tcp [ 2007.735188] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:15:00 (1789582500) [ 2010.016153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg438-server [ 2012.547795] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:15:04 (1789582504) [ 2014.179923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg438-server [ 2016.645613] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:15:09 (1789582509) [ 2018.873994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg438-server [ 2020.836431] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:15:13 (1789582513) [ 2020.943505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.986645] LNet: 94580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.998142] LNetError: 94580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.004266] LNet: Removed LNI 192.168.204.38@tcp [ 2021.027254] LNet: Removed LNI 192.168.204.38@tcp1 [ 2021.045333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.134659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.171488] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2022.177877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2022.290885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2022.343907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2022.577292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2022.637497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2022.747246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2028.141936] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:15:21 (1789582521) [ 2028.260207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.290406] LNet: 95326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.300531] LNetError: 95326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.314726] LNetError: 95326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2028.347953] LNet: Removed LNI 192.168.204.38@tcp [ 2028.373762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.430776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2028.455485] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2035.920075] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:15:29 (1789582529) [ 2036.194747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.234511] LNet: 96039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.249824] LNetError: 96039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.268856] LNet: Removed LNI 192.168.204.38@tcp [ 2036.319553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.400700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2036.452871] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2044.187765] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:15:36 (1789582536) [ 2045.821897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2047.428830] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:15:40 (1789582540) [ 2048.016432] LNet: 97089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.029415] LNetError: 97089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.069293] LNet: Removed LNI 192.168.204.38@tcp [ 2048.647964] Key type .llcrypt unregistered [ 2048.651208] Key type ._llcrypt unregistered [ 2058.603675] Key type ._llcrypt registered [ 2058.605325] Key type .llcrypt registered [ 2059.131521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2059.202328] alg: No test for adler32 (adler32-zlib) [ 2060.362382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2060.478606] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2071.492602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 2080.005251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.138@tcp [ 2080.534423] LNet: 97509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.548538] LNetError: 97509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.569508] LNet: Removed LNI 192.168.204.38@tcp [ 2081.060526] Key type .llcrypt unregistered [ 2081.062257] Key type ._llcrypt unregistered [ 2092.630647] Key type ._llcrypt registered [ 2092.631959] Key type .llcrypt registered [ 2093.671425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2095.277790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2097.175466] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:16:29 (1789582589) [ 2098.703669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2100.649811] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:16:33 (1789582593) [ 2102.389952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg438-server [ 2104.491326] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:16:37 (1789582597) [ 2106.415081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg438-server [ 2108.411233] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:16:41 (1789582601) [ 2110.281075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.296352] alg: No test for adler32 (adler32-zlib) [ 2111.437296] Lustre: Lustre: Build Version: 2.17.58_39_g2c5a7bf [ 2111.675069] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2113.385238] Key type lgssc registered [ 2114.548764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2136.826825] Key type lgssc unregistered [ 2137.178123] LNet: 100748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.200894] LNetError: 100748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.241311] LNet: Removed LNI 192.168.204.38@tcp [ 2138.254534] Key type .llcrypt unregistered [ 2138.262400] Key type ._llcrypt unregistered [ 2157.012241] Key type ._llcrypt registered [ 2157.015334] Key type .llcrypt registered [ 2158.518538] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:17:31 (1789582651) [ 2158.921925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2158.941194] alg: No test for adler32 (adler32-zlib) [ 2159.927306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.995632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2160.049421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2160.105698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2160.167868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2167.641694] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:17:40 (1789582660) [ 2167.729497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2167.764127] LNet: 102171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.797871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.880364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2167.910369] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2168.971629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.988397] LNet: 102323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.000426] LNetError: 102323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.028561] LNet: Removed LNI 192.168.204.38@tcp [ 2169.101294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.183493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2169.251461] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2169.783710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.802787] LNetError: 102351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.818382] LNet: Removed LNI 192.168.204.38@tcp [ 2169.836217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.882420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2169.925530] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2176.792903] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:17:49 (1789582669) [ 2176.857506] LNet: 102941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.863130] LNet: 102941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2176.870220] LNetError: 102941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.893224] LNet: Removed LNI 192.168.204.38@tcp [ 2177.594329] Key type .llcrypt unregistered [ 2177.597889] Key type ._llcrypt unregistered [ 2183.938426] Key type ._llcrypt registered [ 2183.940785] Key type .llcrypt registered [ 2184.156634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2184.180769] alg: No test for adler32 (adler32-zlib) [ 2185.123736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.206092] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2185.208716] LNet: *** cfs_fail_loc=e003, val=100*** [ 2186.183491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2186.208170] LNet: *** cfs_fail_loc=e003, val=100*** [ 2186.213736] LNet: Skipped 2 previous similar messages [ 2186.226754] Ping reply received with mapping: [ 2186.234169] found metadata reply NI[192.168.204.38@tcp] [ 2186.241713] 26 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2186.250774] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2186.257966] 5a 5a 5a 5a 5a 5a 5a 5a [ 2186.443617] LNet: 103346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.457893] LNetError: 103346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.479719] LNet: Removed LNI 192.168.204.38@tcp [ 2187.127789] Key type .llcrypt unregistered [ 2187.133866] Key type ._llcrypt unregistered [ 2197.995931] Key type ._llcrypt registered [ 2198.002024] Key type .llcrypt registered [ 2199.586749] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:18:12 (1789582692) [ 2200.153359] Key type .llcrypt unregistered [ 2200.159504] Key type ._llcrypt unregistered [ 2208.289963] Key type ._llcrypt registered [ 2208.295658] Key type .llcrypt registered [ 2208.758629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2208.823026] alg: No test for adler32 (adler32-zlib) [ 2210.305554] LNetError: 104396:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2210.864507] Key type .llcrypt unregistered [ 2210.869376] Key type ._llcrypt unregistered [ 2218.308124] Key type ._llcrypt registered [ 2218.314058] Key type .llcrypt registered [ 2219.975343] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:18:32 (1789582712) [ 2220.778490] Key type .llcrypt unregistered [ 2220.780937] Key type ._llcrypt unregistered [ 2228.203650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2228.211949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2230.812226] Key type ._llcrypt registered [ 2230.815361] Key type .llcrypt registered [ 2231.084273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2231.104478] alg: No test for adler32 (adler32-zlib) [ 2232.164783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.295565] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2242.041303] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:18:55 (1789582735) [ 2242.360725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.403442] LNet: 106188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.412758] LNetError: 106188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2242.437491] LNet: Removed LNI 10.1.2.3@tcp [ 2242.488673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.316335] Key type .llcrypt unregistered [ 2243.320241] Key type ._llcrypt unregistered [ 2249.824837] Key type ._llcrypt registered [ 2249.828435] Key type .llcrypt registered [ 2249.993531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2250.052033] alg: No test for adler32 (adler32-zlib) [ 2251.054324] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2251.129765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2251.338400] LNet: 106476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.349955] LNetError: 106476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.391505] LNet: Removed LNI 192.168.204.38@tcp [ 2253.164113] Key type .llcrypt unregistered [ 2253.172677] Key type ._llcrypt unregistered [ 2266.990371] Key type ._llcrypt registered [ 2266.992436] Key type .llcrypt registered [ 2268.773617] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:19:21 (1789582761) [ 2269.549417] Key type .llcrypt unregistered [ 2269.561421] Key type ._llcrypt unregistered [ 2276.377695] Key type ._llcrypt registered [ 2276.380658] Key type .llcrypt registered [ 2276.632801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2276.647544] alg: No test for adler32 (adler32-zlib) [ 2283.486366] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:19:36 (1789582776) [ 2283.865539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.896340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.944670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2283.963848] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2284.456371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2284.478413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2284.504530] LNet: Removed LNI 192.168.204.38@tcp [ 2284.671685] LNet: 108237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.014431] Key type .llcrypt unregistered [ 2285.017331] Key type ._llcrypt unregistered [ 2295.287559] Key type ._llcrypt registered [ 2295.289707] Key type .llcrypt registered [ 2297.315510] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:19:49 (1789582789) [ 2297.887036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.908468] alg: No test for adler32 (adler32-zlib) [ 2298.944467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.049346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.085201] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2300.036874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2300.083881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.38@tcp [ 2300.104385] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2300.108511] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2300.111483] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2300, 0, 0 [ 2300.119274] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2300.123621] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2300.128773] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2300.133633] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2300, 0, 0 [ 2300.141617] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2300.146400] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2300.163856] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2300.170499] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2300, 0, 2350 [ 2300.177682] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2300.183143] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2300.190772] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2300.196408] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2300, 0, 2350 [ 2300.204652] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2307.500903] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:20:00 (1789582800) [ 2308.284227] LNet: 109795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.291204] LNetError: 109795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.310833] LNet: Removed LNI 192.168.204.38@tcp [ 2309.090601] Key type .llcrypt unregistered [ 2309.098902] Key type ._llcrypt unregistered [ 2315.392208] Key type ._llcrypt registered [ 2315.394516] Key type .llcrypt registered [ 2315.612589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2315.625666] alg: No test for adler32 (adler32-zlib) [ 2316.542984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2316.575348] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2323.875345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 2330.361626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.427154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2330.443889] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 2331.325701] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2331.537865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2331.540154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2331.554859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2334.033420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2334.077614] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2346.952815] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:20:39 (1789582839) [ 2347.093641] LNet: 111304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.106849] LNetError: 111304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.114373] LNet: Removed LNI 192.168.204.38@tcp [ 2348.128251] LNet: Removed LNI 192.168.204.38@tcp1 [ 2348.132816] LNetError: 111304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2348.153789] LNetError: 111304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2348.202283] LNet: Removed LNI 10.1.2.3@tcp99 [ 2348.993659] Key type .llcrypt unregistered [ 2348.996827] Key type ._llcrypt unregistered [ 2357.007043] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2357.011520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2359.614722] Key type ._llcrypt registered [ 2359.618557] Key type .llcrypt registered [ 2359.783948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2359.803553] alg: No test for adler32 (adler32-zlib) [ 2360.714977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.780039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.803095] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2361.458782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2361.489083] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2363.946395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2363.990112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2364.032335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2364.049235] LNetError: 112052:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2364.094420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2364.170773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2364.678615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2364.810328] LNet: 112072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.818603] LNetError: 112072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2364.829350] LNet: Removed LNI 10.1.2.3@tcp [ 2365.870870] LNet: Removed LNI 192.168.204.38@tcp [ 2366.548616] Key type .llcrypt unregistered [ 2366.552462] Key type ._llcrypt unregistered [ 2379.108393] Key type ._llcrypt registered [ 2379.110644] Key type .llcrypt registered [ 2380.367711] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:21:13 (1789582873) [ 2380.686218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.698186] alg: No test for adler32 (adler32-zlib) [ 2381.720408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.799857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2381.857973] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2383.034521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2383.163646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg438-client@tcp [ 2391.287496] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:21:23 (1789582883) [ 2391.491958] LNet: 113615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.505700] LNetError: 113615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.532905] LNet: Removed LNI 192.168.204.38@tcp [ 2392.182590] Key type .llcrypt unregistered [ 2392.190455] Key type ._llcrypt unregistered [ 2399.176683] Key type ._llcrypt registered [ 2399.179964] Key type .llcrypt registered [ 2399.435783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2399.468453] alg: No test for adler32 (adler32-zlib) [ 2400.414026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2400.444616] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2400.484220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2407.555783] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:21:40 (1789582900) [ 2409.419462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2411.326621] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:21:44 (1789582904) [ 2413.033731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2414.605313] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:21:47 (1789582907) [ 2414.844780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2414.872543] LNet: 115060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.877619] LNetError: 115060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.893192] LNet: Removed LNI 192.168.204.38@tcp [ 2414.914256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.962571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2414.978174] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2415.861298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2415.949399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2422.341982] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:21:55 (1789582915) [ 2422.605235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.642536] LNet: 115777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.651206] LNetError: 115777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.669177] LNet: Removed LNI 192.168.204.38@tcp [ 2422.703478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.769786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2422.829563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2423.435231] Key type .llcrypt unregistered [ 2423.436838] Key type ._llcrypt unregistered [ 2436.936500] Key type ._llcrypt registered [ 2436.941416] Key type .llcrypt registered [ 2438.605667] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:22:11 (1789582931) [ 2439.017096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2439.039845] alg: No test for adler32 (adler32-zlib) [ 2440.087373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.163325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2440.191876] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2441.302480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2441.372159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2448.433897] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:22:21 (1789582941) [ 2448.588224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2448.625731] LNet: 117346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.645797] LNetError: 117346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2449.704234] LNet: Removed LNI 192.168.204.38@tcp [ 2449.743306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.839498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2449.917463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2456.709201] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:22:29 (1789582949) [ 2456.797375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.830229] LNet: 117929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.996951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2457.327655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2457.355320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2457.396180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2457.678707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2457.718297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2457.779321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2457.816120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.895638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2458.300996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2458.338513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2458.396808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2458.721578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2458.751471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2465.717994] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:22:38 (1789582958) [ 2465.815361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.847496] LNet: 118573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2465.856229] LNet: 118573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2465.884264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2465.991815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2466.021836] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2466.797465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2466.812405] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2467.590495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2467.674302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2467.916606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.939497] LNet: 118857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.950864] LNetError: 118857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.957628] LNet: Removed LNI 192.168.204.38@tcp [ 2467.978395] LNet: Removed LNI 192.168.204.38@tcp2 [ 2467.997068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.059939] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2468.068356] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2468.074384] LNet: 118863:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2468.832348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2469.095308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2469.135650] LNetError: 118900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.143127] LNetError: 118900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2469.149431] LNet: Removed LNI 192.168.204.38@tcp [ 2469.162760] LNet: Removed LNI 192.168.204.38@tcp2 [ 2469.191193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.254160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2469.298623] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2470.194707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2470.218806] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2471.024059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2471.080464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2471.158284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2471.419303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.485985] LNetError: 119185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.493398] LNetError: 119185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2471.499776] LNet: Removed LNI 192.168.204.38@tcp [ 2472.552213] LNet: Removed LNI 192.168.204.38@tcp2 [ 2472.603918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.714881] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2472.720643] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2472.724581] LNet: 119191:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2473.031225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2473.202939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.219840] LNet: 119228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.225200] LNet: 119228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2473.231867] LNet: Removed LNI 192.168.204.38@tcp [ 2474.280449] LNet: Removed LNI 192.168.204.38@tcp2 [ 2474.298989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.338347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.357805] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2475.187538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2475.217029] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2476.446647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2476.537448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2476.814544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.841760] LNetError: 119507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.851710] LNetError: 119507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2476.863649] LNet: Removed LNI 192.168.204.38@tcp [ 2476.886922] LNet: Removed LNI 192.168.204.38@tcp2 [ 2476.923761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2477.011291] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2477.016584] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2477.022932] LNet: 119513:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2477.747972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2477.970689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.008220] LNet: Removed LNI 192.168.204.38@tcp [ 2479.018018] LNet: Removed LNI 192.168.204.38@tcp2 [ 2479.047358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.149364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2479.180121] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2480.159766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2480.205408] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2481.388703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2481.440413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2481.506693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2481.672217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.693182] LNet: 119835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.699679] LNet: 119835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2482.720176] LNet: Removed LNI 192.168.204.38@tcp [ 2483.753852] LNet: Removed LNI 192.168.204.38@tcp2 [ 2483.813167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2483.936789] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2483.952764] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2483.967840] LNet: 119841:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2484.538441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2484.777190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.823758] LNet: Removed LNI 192.168.204.38@tcp [ 2485.864670] LNet: Removed LNI 192.168.204.38@tcp2 [ 2485.889957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2485.970657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.012250] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2486.699679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2486.721198] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2487.797340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2487.852481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2487.926549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2488.304902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.333646] LNetError: 120175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2488.341434] LNetError: 120175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2488.349660] LNet: Removed LNI 192.168.204.38@tcp [ 2488.361880] LNet: Removed LNI 192.168.204.38@tcp2 [ 2488.387703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.460495] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2488.467953] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2488.477793] LNet: 120181:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2489.111298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2489.508562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2490.593082] LNet: Removed LNI 192.168.204.38@tcp [ 2490.608564] LNet: Removed LNI 192.168.204.38@tcp2 [ 2490.651596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.730768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2490.767849] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2491.613459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2491.656098] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2492.639075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2492.690906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2492.725745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2492.810401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2493.155941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.240199] LNet: Removed LNI 192.168.204.38@tcp [ 2495.276216] LNet: Removed LNI 192.168.204.38@tcp2 [ 2495.367810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.465222] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2495.478666] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2496.212138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2503.239378] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2504.868226] Lustre: DEBUG MARKER: == sanity-lnet test 411: No segfault on unmatched quotes in lnetctl input ========================================================== 14:23:17 (1789582997) [ 2505.056680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2505.106658] LNet: 121311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2505.123118] LNet: 121311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2505.129448] LNetError: 121311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2505.142254] LNetError: 121311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2505.155686] LNet: Removed LNI 192.168.204.38@tcp [ 2506.224703] LNet: Removed LNI 192.168.204.38@tcp2 [ 2506.250073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.340700] LNetError: 121318:0:(config.c:1725:lnet_inet_select()) ksocklnd: failed to find UP interface lnd tunables [ 2506.365564] LNetError: Error -100 starting up LNI tcp [ 2512.088872] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:23:25 (1789583005) [ 2512.263117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2512.296297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2512.577198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2512.599550] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2513.341842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2513.384957] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2514.420463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2514.470116] LNet: Removed LNI 10.1.2.3@tcp [ 2514.478634] LNet: Removed LNI 192.168.204.38@tcp [ 2514.495939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.536385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2514.582718] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2514.594746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2514.911723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2514.931809] LNet: Removed LNI 192.168.204.38@tcp [ 2514.943740] LNet: Removed LNI 10.1.2.3@tcp [ 2521.257263] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 14:23:34 (1789583014) [ 2521.444854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.488583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2521.554895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2521.591107] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2522.359852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2522.389553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2522.428651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.468488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.509395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.555440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.611076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.665442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.735947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.777035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.817441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2522.853532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2523.077795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2529.657774] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 14:23:42 (1789583022) [ 2529.866223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2530.982859] LNet: Removed LNI 192.168.204.38@tcp [ 2531.032464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2531.116174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2531.154427] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2531.976634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2532.043092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2532.139069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.185644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.240044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.324766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.365583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.406390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.459256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.513626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.554887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.588426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.743798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2532.792788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.864294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.907599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2532.979425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.040803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.158623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.208809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.249743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.277852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2533.329840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.38@tcp [ 2540.343710] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:23:53 (1789583033) [ 2540.520360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2540.578995] LNet: 124305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.587052] LNet: 124305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2540.592760] LNetError: 124305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.603766] LNetError: 124305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2540.617150] LNet: Removed LNI 192.168.204.38@tcp [ 2540.664743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2540.796686] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2541.021676] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2541.026992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2541.034996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2543.416984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2543.445593] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2544.473019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2544.497068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2546.674946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2546.735151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2546.789510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.38@tcp -m PUT -e local_timeout -r 1 [ 2546.850653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2546.931886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2546.935880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2547.178688] LNetError: 124369:0:(lib-move.c:4794:lnet_parse()) 192.168.204.38@tcp, src 192.168.204.38@tcp: Dropping PUT (error -108 looking up sender) [ 2548.193342] LNet: Removed LNI 10.1.2.3@tcp [ 2549.223275] LNet: Removed LNI 192.168.204.38@tcp [ 2549.733164] Key type .llcrypt unregistered [ 2549.737556] Key type ._llcrypt unregistered [ 2562.541771] Key type ._llcrypt registered [ 2562.543773] Key type .llcrypt registered [ 2563.892552] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:24:16 (1789583056) [ 2564.281373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.302279] alg: No test for adler32 (adler32-zlib) [ 2565.369637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.879231] LNet: 125700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2566.586234] Key type .llcrypt unregistered [ 2566.593521] Key type ._llcrypt unregistered [ 2574.372230] Key type ._llcrypt registered [ 2574.376237] Key type .llcrypt registered [ 2574.588468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2574.608387] alg: No test for adler32 (adler32-zlib) [ 2575.566842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2575.593313] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2583.205831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 2583.431604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2584.079924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.138@tcp [ 2584.356086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2584.396478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2584.435252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2584.496450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2584.520583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2584.539294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2584.565087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2584.617083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2584.650166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2584.665206] LNet: 126152:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2584.678699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2584.704918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2584.734390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2584.772655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2584.800087] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.138@tcp [ 2584.812424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2584.846853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2585.667693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2585.925206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2585.952841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2585.977213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2585.999612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2586.020916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2586.047070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2586.083795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2586.133536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2586.179706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2586.250799] LNet: 126271:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2586.285702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2586.325454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2586.357906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2586.398969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2586.446711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2586.512623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2587.181491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.138@tcp [ 2587.544773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2587.596986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2587.625878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2587.648229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2587.671088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2587.693420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2587.719246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2587.740910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2587.757474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2587.773620] LNet: 126391:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2587.789407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2587.806065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2587.827757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2587.848569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2587.882171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2587.913045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2588.420379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2588.721771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2588.754543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2588.813485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2588.857814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2588.894349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2588.921689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2588.956599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2588.993520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2589.030161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2589.052850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2589.083177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2589.112545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2589.137813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2589.159556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2589.182534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2589.512617] LNet: 126556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.522076] LNetError: 126556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.537467] LNet: Removed LNI 192.168.204.38@tcp [ 2590.061655] Key type .llcrypt unregistered [ 2590.063939] Key type ._llcrypt unregistered [ 2608.466244] Key type ._llcrypt registered [ 2608.475026] Key type .llcrypt registered [ 2609.686644] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:25:02 (1789583102) [ 2610.116678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.139919] alg: No test for adler32 (adler32-zlib) [ 2611.139880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2611.729838] LNet: 127453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.375514] Key type .llcrypt unregistered [ 2612.379232] Key type ._llcrypt unregistered [ 2618.538583] Key type ._llcrypt registered [ 2618.541531] Key type .llcrypt registered [ 2618.860847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2618.876569] alg: No test for adler32 (adler32-zlib) [ 2619.745826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2619.765117] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2627.831150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.138@tcp [ 2636.247208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2636.306347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2636.340932] LNet: Added LNI 192.168.204.38@tcp1 [8/256/0/180] [ 2637.205483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2637.922665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.138@tcp [ 2638.171468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2638.198164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2638.224373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2638.245223] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.138@tcp [ 2638.251244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2638.277868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2638.302774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2638.325308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2638.349679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2638.380984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2638.415873] LNet: 128108:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2638.438297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2638.477888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2638.526762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2638.563554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2638.595233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2638.624807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2639.414332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2639.777801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2639.811277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2639.834898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2639.866615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2639.926153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2639.982750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2640.027237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2640.085036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2640.137462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2640.160861] LNet: 128227:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2640.181681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2640.242881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2640.285315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2640.321211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2640.356305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2640.405074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2641.139569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.138@tcp [ 2641.454191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2641.509058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2641.562525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2641.619391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2641.663611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2641.696259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2641.739626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2641.779647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2641.813531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2641.833814] LNet: 128346:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2641.859473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2641.906026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2641.953114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2642.000394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2642.043945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2642.100969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2642.779728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2643.001745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2643.038356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2643.081691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2643.136754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2643.169124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2643.210867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2643.265771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2643.328117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2643.391981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2643.450792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2643.515356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2643.590201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2643.663614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2643.731658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2643.779188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2649.320616] LNet: 128547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2649.325910] LNetError: 128547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.336363] LNet: Removed LNI 192.168.204.38@tcp [ 2650.344877] LNetError: 128547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.360423] LNet: Removed LNI 192.168.204.38@tcp1 [ 2650.797323] Key type .llcrypt unregistered [ 2650.802914] Key type ._llcrypt unregistered [ 2668.341993] Key type ._llcrypt registered [ 2668.344369] Key type .llcrypt registered [ 2669.608807] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 14:26:02 (1789583162) [ 2670.072309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2670.093848] alg: No test for adler32 (adler32-zlib) [ 2670.975535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2671.013792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2671.034837] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2671.775290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2671.805536] LNet: Added LNI 192.168.204.38@tcp2 [8/256/0/180] [ 2672.675835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.38@tcp [ 2672.769241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2673.826137] LNet: Removed LNI 192.168.204.38@tcp [ 2680.934288] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:26:13 (1789583173) [ 2683.193496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2686.650689] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:26:18 (1789583178) [ 2698.904039] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:26:30 (1789583190) [ 2707.798785] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:26:40 (1789583200) [ 2715.586626] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:26:48 (1789583208) [ 2722.643086] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:26:55 (1789583215) [ 2730.208379] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:27:03 (1789583223) [ 2737.412564] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:27:10 (1789583230) [ 2744.672883] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:27:17 (1789583237) [ 2752.715464] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:27:25 (1789583245) [ 2761.331765] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:27:33 (1789583253) [ 2769.351958] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:27:42 (1789583262) [ 2777.771621] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:27:50 (1789583270) [ 2784.938896] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:27:57 (1789583277) [ 2792.907651] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:28:05 (1789583285) [ 2799.715219] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:28:12 (1789583292) [ 2807.545602] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:28:20 (1789583300) [ 2815.934539] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:28:28 (1789583308) [ 2824.858949] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:28:37 (1789583317) [ 2825.023456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2825.050320] LNet: 140397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2825.063986] LNetError: 140397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2825.100454] LNet: Removed LNI 192.168.204.38@tcp2 [ 2825.171713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2825.386664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2825.436965] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2826.742657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2826.764244] LNet: 140544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2826.778344] LNetError: 140544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.797558] LNet: Removed LNI 192.168.204.38@tcp [ 2826.822597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2826.886711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2826.915211] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2834.764494] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:28:47 (1789583327) [ 2843.985563] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:28:56 (1789583336) [ 2851.857765] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:29:04 (1789583344) [ 2859.503191] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:29:12 (1789583352) [ 2867.496795] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:29:20 (1789583360) [ 2875.245389] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:29:28 (1789583368) [ 2882.998150] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:29:35 (1789583375) [ 2889.953396] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:29:43 (1789583383) [ 2898.297579] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:29:50 (1789583390) [ 2906.706528] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:29:59 (1789583399) [ 2915.657816] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:30:08 (1789583408) [ 2924.014860] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:30:16 (1789583416) [ 2932.873656] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2664 sec ========== 14:30:25 (1789583425) [ 2933.168474] LNet: 148115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.175983] LNetError: 148115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.198099] LNet: Removed LNI 192.168.204.38@tcp [ 2934.057587] Key type .llcrypt unregistered [ 2934.064726] Key type ._llcrypt unregistered [ 2950.889637] Key type ._llcrypt registered [ 2950.906289] Key type .llcrypt registered [ 2951.646248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2951.662686] alg: No test for adler32 (adler32-zlib) [ 2952.864394] Lustre: Lustre: Build Version: 2.17.58_39_g2c5a7bf [ 2953.240341] LNet: Added LNI 192.168.204.38@tcp [8/256/0/180] [ 2954.928399] Key type lgssc registered [ 2956.322980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3064.383443] Lustre: Mounted lustre-client - version 2.17.58_39_g2c5a7bf [ 3069.882708] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3077.749757] Lustre: Unmounted lustre-client [ 3138.000288] Key type lgssc unregistered [ 3138.375835] LNet: 151648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3138.386168] LNetError: 151648:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3138.427283] LNet: Removed LNI 192.168.204.38@tcp [ 3139.297182] Key type .llcrypt unregistered [ 3139.300953] Key type ._llcrypt unregistered