[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505385928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003274] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007366] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011060] Yama: becoming mindful. [ 0.012051] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.022378] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026804] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029122] *** VALIDATE tmpfs *** [ 0.031313] *** VALIDATE proc *** [ 0.032242] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034301] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037030] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040701] debug: unmapping init [mem 0xffffffffaf659000-0xffffffffaf660fff] [ 0.043256] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044821] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045029] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048010] ... value mask: 0000ffffffffffff [ 0.049016] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052314] rcu: Hierarchical SRCU implementation. [ 0.054505] smp: Bringing up secondary CPUs ... [ 0.055562] x86: Booting SMP configuration: [ 0.056020] .... node #0, CPUs: #1 #2 #3 [ 0.059218] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131658] node 0 deferred pages initialised in 67ms [ 0.135095] devtmpfs: initialized [ 0.135785] x86/mm: Memory block size: 128MB [ 0.137217] gcov: version magic: 0x41383552 [ 0.140160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144303] pinctrl core: initialized pinctrl subsystem [ 0.145123] [ 0.145512] ************************************************************* [ 0.147012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149012] ** ** [ 0.151010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152008] ** ** [ 0.154010] ** This means that this kernel is built to expose internal ** [ 0.155062] ** IOMMU data structures, which may compromise security on ** [ 0.156006] ** your system. ** [ 0.158006] ** ** [ 0.159005] ** If you see this message and you are not debugging the ** [ 0.160006] ** kernel, report this immediately to your vendor! ** [ 0.161006] ** ** [ 0.163005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164006] ************************************************************* [ 0.165832] NET: Registered protocol family 16 [ 0.167337] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.169148] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173079] cpuidle: using governor menu [ 0.174555] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175306] PCI: Using configuration type 1 for base access [ 0.176086] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181112] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183297] cryptd: max_cpu_qlen set to 1000 [ 0.186020] ACPI: Added _OSI(Module Device) [ 0.188012] ACPI: Added _OSI(Processor Device) [ 0.189013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196206] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201375] ACPI: Interpreter enabled [ 0.203058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205013] ACPI: Using IOAPIC for interrupt routing [ 0.206100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209413] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.223018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.231413] acpiphp: Slot [2] registered [ 0.233081] acpiphp: Slot [5] registered [ 0.234119] acpiphp: Slot [6] registered [ 0.235098] acpiphp: Slot [3] registered [ 0.237102] acpiphp: Slot [4] registered [ 0.237953] acpiphp: Slot [7] registered [ 0.238053] acpiphp: Slot [8] registered [ 0.238952] acpiphp: Slot [9] registered [ 0.240088] acpiphp: Slot [10] registered [ 0.240833] acpiphp: Slot [11] registered [ 0.242053] acpiphp: Slot [12] registered [ 0.242893] acpiphp: Slot [13] registered [ 0.243094] acpiphp: Slot [14] registered [ 0.243897] acpiphp: Slot [15] registered [ 0.245055] acpiphp: Slot [16] registered [ 0.246088] acpiphp: Slot [17] registered [ 0.247051] acpiphp: Slot [18] registered [ 0.247848] acpiphp: Slot [19] registered [ 0.248074] acpiphp: Slot [20] registered [ 0.249027] acpiphp: Slot [21] registered [ 0.249836] acpiphp: Slot [22] registered [ 0.251129] acpiphp: Slot [23] registered [ 0.253097] acpiphp: Slot [24] registered [ 0.254106] acpiphp: Slot [25] registered [ 0.255084] acpiphp: Slot [26] registered [ 0.257148] acpiphp: Slot [27] registered [ 0.258103] acpiphp: Slot [28] registered [ 0.260157] acpiphp: Slot [29] registered [ 0.261153] acpiphp: Slot [30] registered [ 0.263179] acpiphp: Slot [31] registered [ 0.264120] PCI host bridge to bus 0000:00 [ 0.266015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.267011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.268010] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.270011] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.271010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.272015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.273111] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.275713] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.278118] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.285873] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.288442] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.289013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.291017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.293013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.294482] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.296591] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.298026] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.299623] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.303010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.310013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.313566] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.317949] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.324022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.330019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.342018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.350524] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.356022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.367035] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.383015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.392479] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.394280] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.396324] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.398410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.401247] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.406094] iommu: Default domain type: Passthrough [ 0.408514] SCSI subsystem initialized [ 0.410133] ACPI: bus type USB registered [ 0.411056] usbcore: registered new interface driver usbfs [ 0.412058] usbcore: registered new interface driver hub [ 0.413059] usbcore: registered new device driver usb [ 0.414163] pps_core: LinuxPPS API ver. 1 registered [ 0.415006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.417057] PTP clock support registered [ 0.419121] EDAC MC: Ver: 3.0.0 [ 0.420278] PCI: Using ACPI for IRQ routing [ 0.421678] NetLabel: Initializing [ 0.422007] NetLabel: domain hash size = 128 [ 0.423010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.424120] NetLabel: unlabeled traffic allowed by default [ 0.425216] vgaarb: loaded [ 0.426252] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.428010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.435400] clocksource: Switched to clocksource kvm-clock [ 0.538423] VFS: Disk quotas dquot_6.6.0 [ 0.539428] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.541338] *** VALIDATE ramfs *** [ 0.542422] *** VALIDATE hugetlbfs *** [ 0.543832] pnp: PnP ACPI init [ 0.545579] pnp: PnP ACPI: found 6 devices [ 0.563955] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.565763] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.566862] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.567957] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.569153] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.570454] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.572078] NET: Registered protocol family 2 [ 0.573652] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.576567] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.579411] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.584586] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.587879] TCP: Hash tables configured (established 65536 bind 65536) [ 0.589836] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.592364] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.594346] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.596231] NET: Registered protocol family 1 [ 0.597793] RPC: Registered named UNIX socket transport module. [ 0.598935] RPC: Registered udp transport module. [ 0.599911] RPC: Registered tcp transport module. [ 0.601058] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.602279] NET: Registered protocol family 44 [ 0.603176] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.604317] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.606235] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.607814] PCI: CLS 0 bytes, default 64 [ 0.608745] Unpacking initramfs... [ 1.992180] debug: unmapping init [mem 0xffffa04d7cc64000-0xffffa04d7ffcffff] [ 1.997462] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.000757] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.004804] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.514439] Initialise system trusted keyrings [ 2.516581] Key type blacklist registered [ 2.518291] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.526183] zbud: loaded [ 2.529430] *** VALIDATE nfs *** [ 2.530726] *** VALIDATE nfs4 *** [ 2.532431] pstore: using deflate compression [ 2.536389] Platform Keyring initialized [ 2.621629] NET: Registered protocol family 38 [ 2.626750] Key type asymmetric registered [ 2.628368] Asymmetric key parser 'x509' registered [ 2.630353] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.633648] io scheduler mq-deadline registered [ 2.636082] io scheduler kyber registered [ 2.638376] io scheduler bfq registered [ 2.640680] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.643857] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.646623] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.649719] ACPI: Power Button [PWRF] [ 2.654497] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.659875] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.668080] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.694825] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.721881] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.727022] Non-volatile memory driver v1.3 [ 2.728600] Linux agpgart interface v0.103 [ 2.758445] virtio_blk virtio1: [vda] 150600 512-byte logical blocks (77.1 MB/73.5 MiB) [ 2.761086] vda: detected capacity change from 0 to 77107200 [ 2.773538] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.776110] vdb: detected capacity change from 0 to 1073741824 [ 2.781820] libphy: Fixed MDIO Bus: probed [ 2.785873] usbcore: registered new interface driver usbserial_generic [ 2.788035] usbserial: USB Serial support registered for generic [ 2.790338] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.794594] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.796436] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.798911] mousedev: PS/2 mouse device common for all mice [ 2.801781] rtc_cmos 00:05: RTC can wake from S4 [ 2.805252] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.805527] rtc_cmos 00:05: registered as rtc0 [ 2.810493] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.813474] intel_pstate: CPU model not supported [ 2.813764] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.819176] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.819763] hid: raw HID events driver (C) Jiri Kosina [ 2.823753] usbcore: registered new interface driver usbhid [ 2.825392] usbhid: USB HID core driver [ 2.826943] drop_monitor: Initializing network drop monitor service [ 2.829033] Initializing XFRM netlink socket [ 2.830353] NET: Registered protocol family 10 [ 2.832388] Segment Routing with IPv6 [ 2.834036] NET: Registered protocol family 17 [ 2.836025] mpls_gso: MPLS GSO support [ 2.840030] RAS: Correctable Errors collector initialized. [ 2.841117] AVX version of gcm_enc/dec engaged. [ 2.841980] AES CTR mode by8 optimization enabled [ 2.901391] sched_clock: Marking stable (2901371360, 0)->(3766258251, -864886891) [ 2.904663] registered taskstats version 1 [ 2.906479] Loading compiled-in X.509 certificates [ 2.909463] zswap: loaded using pool lzo/zbud [ 2.931727] Key type big_key registered [ 2.943230] Key type encrypted registered [ 2.944245] ima: No TPM chip found, activating TPM-bypass! [ 2.945582] ima: Allocated hash algorithm: sha1 [ 2.946749] ima: No architecture policies found [ 2.947949] evm: Initialising EVM extended attributes: [ 2.949184] evm: security.selinux [ 2.950120] evm: security.ima [ 2.950831] evm: security.capability [ 2.951623] evm: HMAC attrs: 0x1 [ 2.953466] rtc_cmos 00:05: setting system clock to 2026-09-29 14:24:08 UTC (1790691848) [ 2.958393] debug: unmapping init [mem 0xffffffffb0603000-0xffffffffb07fffff] [ 2.960340] debug: unmapping init [mem 0xffffffffaf382000-0xffffffffaf658fff] [ 2.968057] Write protecting the kernel read-only data: 28672k [ 2.971060] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 2.973084] debug: unmapping init [mem 0xffffffffae314000-0xffffffffae3fffff] [ 3.000817] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.005560] systemd[1]: Detected virtualization kvm. [ 3.006789] systemd[1]: Detected architecture x86-64. [ 3.007766] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.034485] systemd[1]: No hostname configured. [ 3.036266] systemd[1]: Set hostname to . [ 3.038258] random: systemd: uninitialized urandom read (16 bytes read) [ 3.040612] systemd[1]: Initializing machine ID from random generator. [ 3.138577] random: systemd: uninitialized urandom read (16 bytes read) [ 3.140534] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.143842] random: systemd: uninitialized urandom read (16 bytes read) [ 3.145583] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.149589] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.649230] device-mapper: uevent: version 1.0.3 [ 3.651357] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.258757] virtio_net virtio0 ens2: renamed from eth0 [ 4.339493] scsi host0: ata_piix [ 4.375023] scsi host1: ata_piix [ 4.376793] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.379796] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 5.040030] random: fast init done [ 7.970115] dracut-initqueue[575]: RTNETLINK answers: File exists [ 8.456286] random: crng init done [ 8.457732] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.089299] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.156889] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.393234] SELinux: Disabled at runtime. [ 10.450239] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.457636] systemd[1]: Detected virtualization kvm. [ 10.459205] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 10.892864] systemd[1]: initrd-switch-root.service: Succeeded. [ 10.895415] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 10.901510] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 10.905562] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 10.908945] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 10.916537] systemd[1]: Starting Journal Service... Starting Journal Service... [ 10.934984] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ 10.974610] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.401233] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.660369] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 11.759721] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 11.859922] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 11.874492] EDAC sbridge: Ver: 1.1.2 [ 13.098182] Key type dns_resolver registered [ 13.401595] NFS: Registering the id_resolver key type [ 13.403318] Key type id_resolver registered [ 13.404339] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg148-client login: [ 41.055045] libcfs: loading out-of-tree module taints kernel. [ 41.273403] Key type ._llcrypt registered [ 41.279740] Key type .llcrypt registered [ 41.962908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.986899] alg: No test for adler32 (adler32-zlib) [ 43.308344] Lustre: Lustre: Build Version: 2.17.59_2_g85450b5 [ 44.222801] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 46.008820] Key type lgssc registered [ 47.747553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.950017] hrtimer: interrupt took 2568274 ns [ 176.440029] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5 [ 181.189636] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.253823] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing check_logdir /tmp/testlogs/ [ 201.717910] Lustre: DEBUG MARKER: oleg148-client.virtnet: executing yml_node [ 202.209477] Lustre: lustre-OST0000-osc-ffffa04dc5e57000: disconnect after 23s idle [ 205.474745] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 207.873655] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 210.514332] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 212.326867] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 29 10:27:36 EDT 2026 [ 228.688394] Lustre: Unmounted lustre-client [ 259.252522] Key type lgssc unregistered [ 259.717261] LNet: 8395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.725131] LNetError: 8395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.752733] LNet: Removed LNI 192.168.201.48@tcp [ 260.963143] Key type .llcrypt unregistered [ 260.971149] Key type ._llcrypt unregistered [ 291.250357] Key type ._llcrypt registered [ 291.253254] Key type .llcrypt registered [ 291.371347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 291.380383] alg: No test for adler32 (adler32-zlib) [ 292.214249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 292.251328] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 292.263081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 295.326784] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 295.403368] LNet: 9277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.407694] LNetError: 9277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.431396] LNet: Removed LNI 192.168.201.48@tcp [ 295.988531] Key type .llcrypt unregistered [ 295.991928] Key type ._llcrypt unregistered [ 302.515267] Key type ._llcrypt registered [ 302.518147] Key type .llcrypt registered [ 304.431869] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:29:07 (1790692147) [ 305.080427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 305.089918] alg: No test for adler32 (adler32-zlib) [ 306.110078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.295971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.377844] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.461153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.627819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 306.687279] LNet: 9703:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 319.419911] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:29:22 (1790692162) [ 319.596365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.635628] LNet: 10275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.708981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.916459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 330.140397] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:29:34 (1790692174) [ 330.300685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.352682] LNet: 10866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.415058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.509870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 338.628869] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:29:42 (1790692182) [ 338.969475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.013358] LNet: 11453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.120435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.284925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 346.277275] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:29:50 (1790692190) [ 346.368815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.394283] LNet: 12032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.432476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.516973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 347.049639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 353.317727] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:29:57 (1790692197) [ 353.546079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.597930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.739967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 354.277469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 354.655744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 361.359941] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:30:05 (1790692205) [ 361.534779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.576530] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.595785] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 361.649230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.714789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 368.119565] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:30:12 (1790692212) [ 368.535786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.588814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.640637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 368.667571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.055382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 369.096803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 369.315993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 369.339385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.784864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 369.821127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 370.231680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 370.271889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 370.758460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 370.789480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.302595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 371.392232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.878613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 371.930647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 372.456679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 372.515453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 372.783488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 372.810870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 373.198719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 373.234201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 373.810957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 373.877984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 374.528370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 374.571801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.358660] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:30:25 (1790692225) [ 381.633529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.655532] LNet: 14494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.660291] LNet: 14494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 381.674916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.753726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 381.806646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 389.304294] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:30:33 (1790692233) [ 389.606956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.654370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.722393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 389.757286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 396.662297] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:30:40 (1790692240) [ 396.803568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.857974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.981113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 397.051381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 404.057717] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:30:48 (1790692248) [ 404.179601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.237312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.333642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 404.371440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 411.370587] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:30:55 (1790692255) [ 411.479069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.522147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.610565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.640510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 418.116315] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:31:02 (1790692262) [ 418.222817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.239762] LNet: 17418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.246230] LNet: 17418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 418.273351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.376195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 418.419803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 424.765811] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:31:08 (1790692268) [ 425.066884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.146397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.244695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.708286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.042786] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:31:16 (1790692276) [ 432.122796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.190389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.341596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.542167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.582153] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:31:23 (1790692283) [ 438.746806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.825585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.935828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.099887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.082366] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:31:30 (1790692290) [ 446.262950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.345753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.834301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.132744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.956601] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:31:38 (1790692298) [ 454.217386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.292680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.391541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.446168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.170705] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:31:45 (1790692305) [ 461.420438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.507987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.571690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.604930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.243089] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:31:53 (1790692313) [ 469.461526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.553882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.683848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 475.860968] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:32:00 (1790692320) [ 476.065627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.105100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.176329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 482.297120] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:32:06 (1790692326) [ 482.520292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.555201] LNet: 22861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.565475] LNet: 22861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 482.607617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.706761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 483.172900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 490.028326] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:32:14 (1790692334) [ 490.184861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.251977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.370530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 490.401797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 496.822487] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:32:21 (1790692341) [ 497.020651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.050673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.108319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 497.134993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 503.663204] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:32:27 (1790692347) [ 504.022492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.070573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.128288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 504.158261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 511.178751] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:32:35 (1790692355) [ 511.300236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.335552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.450982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 511.493594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 518.235182] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:32:42 (1790692362) [ 518.379171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.439216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.532902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 518.603528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 518.619925] LNetError: 25808:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 518.651259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 518.679553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 526.661194] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:32:50 (1790692370) [ 527.082502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.135226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.244651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9H7dzzja17 [ 527.382650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9H7dzzja17 [ 527.458725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.9H7dzzja17 [ 534.647514] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:32:58 (1790692378) [ 534.781749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.849686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.948650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 535.001140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 535.076867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 542.407359] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:33:05 (1790692385) [ 542.582313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.699898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.832690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.855475] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 545.358535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 547.543967] Lustre: Lustre: Build Version: 2.17.59_2_g85450b5 [ 549.576655] Key type lgssc registered [ 551.098333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 555.890022] LNetError: 27570:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.48@tcp99 has no available nets [ 558.844191] Key type lgssc unregistered [ 559.176568] LNetError: 28685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.237717] LNet: Removed LNI 192.168.201.48@tcp [ 561.097866] Key type .llcrypt unregistered [ 561.110819] Key type ._llcrypt unregistered [ 567.276737] Key type ._llcrypt registered [ 567.279926] Key type .llcrypt registered [ 568.556756] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:33:32 (1790692412) [ 569.341373] Key type .llcrypt unregistered [ 569.343480] Key type ._llcrypt unregistered [ 577.172275] Key type ._llcrypt registered [ 577.180701] Key type .llcrypt registered [ 577.616742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.661578] alg: No test for adler32 (adler32-zlib) [ 578.589781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.643717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 578.668681] LNet: 29607:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 578.680277] LNet: 29607:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 578.697132] LNet: 29607:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 578.713929] LNet: Message forwarding will be enabled in 2s [ 581.792298] LNet: Message forwarding enabled [ 582.280981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 582.310198] LNet: Message forwarding will stop in 2s to 6s [ 585.890630] LNet: Message forwarding disabled [ 586.883852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 586.907216] LNet: 29706:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 586.919157] LNet: 29706:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 586.939227] LNet: 29706:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 586.961614] LNet: Message forwarding will be enabled in 2s [ 587.170772] LNet: 29719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.929327] Key type .llcrypt unregistered [ 587.932196] Key type ._llcrypt unregistered [ 599.992982] Key type ._llcrypt registered [ 599.995793] Key type .llcrypt registered [ 601.354515] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:34:05 (1790692445) [ 601.864420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.874717] alg: No test for adler32 (adler32-zlib) [ 602.864940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.025858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 609.960948] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:34:13 (1790692453) [ 610.124755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.171697] LNet: 31116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.210803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.323543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 610.356629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 610.389536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 610.425954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 610.462725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 610.495913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 610.531438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 610.585214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 610.629784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 610.666381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 610.720734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 610.823580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 610.873178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 610.901966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 610.922751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 610.946384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 610.971888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 611.004080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.023681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 611.034212] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 617.676793] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:34:21 (1790692461) [ 618.054974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.070191] LNet: 31736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.092225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.159201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.328949] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:34:30 (1790692470) [ 629.548874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg148-server [ 632.007873] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:34:35 (1790692475) [ 632.216390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.304974] LNet: 32665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.337849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.455697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.489903] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 633.711692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[49-53]@tcp [ 633.760423] LNet: 32800:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 633.829326] LNetError: 32671:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.49@tcp is being used as a gateway but routing feature is not turned on [ 633.856419] LNetError: 32671:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.49@tcp has gone from up to down [ 642.413731] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:34:46 (1790692486) [ 644.690898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg148-server [ 646.508653] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:34:50 (1790692490) [ 646.742950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.770632] LNet: 33736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 646.775423] LNetError: 33736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 646.790092] LNet: Removed LNI 192.168.201.48@tcp [ 646.815471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.857751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 646.893129] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 647.600834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[49-54/2]@tcp [ 647.617844] LNet: 33853:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 647.624235] LNet: 33853:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 647.639572] LNetError: 33740:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.49@tcp is being used as a gateway but routing feature is not turned on [ 647.655972] LNetError: 33740:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 647.659082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[49-54/2]@tcp [ 647.671564] LNetError: 33740:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.49@tcp has gone from up to down [ 647.671575] LNetError: 33740:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 654.297954] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:34:58 (1790692498) [ 654.527876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.550566] LNet: 34446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.554967] LNetError: 34446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.577920] LNet: Removed LNI 192.168.201.48@tcp [ 654.610356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.705657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 654.773601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.823183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.911155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.982479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.018224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.074377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 655.174458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.226277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.369409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.483657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.544715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.624631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 655.713136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.765145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.828156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.916697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.955650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.006480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 656.103960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.135205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.192704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.298457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.351287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.403429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 656.494567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.529331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.595681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.666873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.701579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.749514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 656.804816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.846533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.944868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 662.761641] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:35:07 (1790692507) [ 664.491971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg148-server [ 666.122401] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:35:10 (1790692510) [ 668.108742] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg148-server [ 669.637392] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:35:13 (1790692513) [ 669.778383] LNet: 35840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.790096] LNet: 35840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 670.402446] Key type .llcrypt unregistered [ 670.405703] Key type ._llcrypt unregistered [ 676.838418] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 677.138419] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 677.141487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 677.147406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 679.660943] Key type ._llcrypt registered [ 679.662484] Key type .llcrypt registered [ 679.932230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.942799] alg: No test for adler32 (adler32-zlib) [ 680.936506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.055319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.074417] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 681.656553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 681.677482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 683.360929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 683.415742] LNet: Removed LNI 10.1.2.3@tcp [ 683.541168] LNet: 36561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.550966] LNetError: 36561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.581302] LNet: Removed LNI 192.168.201.48@tcp [ 683.992363] Key type .llcrypt unregistered [ 683.993446] Key type ._llcrypt unregistered [ 694.883030] Key type ._llcrypt registered [ 694.884427] Key type .llcrypt registered [ 696.318907] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:35:40 (1790692540) [ 696.806171] Key type .llcrypt unregistered [ 696.807430] Key type ._llcrypt unregistered [ 702.791100] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.892706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.909831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.925053] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.511186] Key type ._llcrypt registered [ 705.514517] Key type .llcrypt registered [ 705.694035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.708504] alg: No test for adler32 (adler32-zlib) [ 706.625408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.729472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.765782] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 707.582791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.599665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.269302] LNet: 38063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.278215] LNetError: 38063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 710.288950] LNet: Removed LNI 10.1.2.3@tcp [ 710.317667] LNet: Removed LNI 192.168.201.48@tcp [ 711.019685] Key type .llcrypt unregistered [ 711.026255] Key type ._llcrypt unregistered [ 722.992654] Key type ._llcrypt registered [ 722.994748] Key type .llcrypt registered [ 724.191516] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:36:08 (1790692568) [ 724.906318] Key type .llcrypt unregistered [ 724.907978] Key type ._llcrypt unregistered [ 731.111839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.346674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.351544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.358289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.910263] Key type ._llcrypt registered [ 733.914174] Key type .llcrypt registered [ 734.150593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 734.160589] alg: No test for adler32 (adler32-zlib) [ 735.138919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.191927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 735.215410] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 737.189429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 737.216856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.038365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 739.064726] LNet: Removed LNI 10.1.2.3@tcp [ 739.082768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 739.122241] LNet: Removed LNI 10.1.2.31@tcp [ 739.141788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 739.167796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.745889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 739.773502] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 740.703184] LNet: 39983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.708081] LNetError: 39983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 740.714291] LNet: Removed LNI 10.1.2.31@tcp [ 740.731935] LNet: Removed LNI 10.1.2.3@tcp [ 741.181583] Key type .llcrypt unregistered [ 741.183628] Key type ._llcrypt unregistered [ 753.990385] Key type ._llcrypt registered [ 753.994600] Key type .llcrypt registered [ 755.304495] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:36:39 (1790692599) [ 755.664160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 755.677715] alg: No test for adler32 (adler32-zlib) [ 756.664973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.706980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.727108] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 757.449368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.463525] LNet: 40934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.467969] LNetError: 40934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.482563] LNet: Removed LNI 192.168.201.48@tcp [ 757.501237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.542332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.48@tcp [ 757.558663] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 758.046485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.48@tcp [ 758.079873] LNet: Removed LNI 192.168.201.48@tcp [ 758.168545] LNet: 41069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.853408] Key type .llcrypt unregistered [ 758.855366] Key type ._llcrypt unregistered [ 770.578992] Key type ._llcrypt registered [ 770.581087] Key type .llcrypt registered [ 771.929284] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:36:56 (1790692616) [ 773.634241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg148-server [ 775.181468] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:36:59 (1790692619) [ 775.698173] Key type .llcrypt unregistered [ 775.699156] Key type ._llcrypt unregistered [ 782.455240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.762774] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.764855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.768065] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 785.305043] Key type ._llcrypt registered [ 785.307543] Key type .llcrypt registered [ 785.565567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.599131] alg: No test for adler32 (adler32-zlib) [ 786.536646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.613401] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 787.405778] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 789.231548] LNet: 42905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.237163] LNetError: 42905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.243911] LNet: Removed LNI 192.168.201.48@tcp1 [ 789.252398] LNet: Removed LNI 10.1.2.3@tcp2 [ 789.859576] Key type .llcrypt unregistered [ 789.863429] Key type ._llcrypt unregistered [ 801.738179] Key type ._llcrypt registered [ 801.740067] Key type .llcrypt registered [ 803.033725] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:37:27 (1790692647) [ 803.356547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.376133] alg: No test for adler32 (adler32-zlib) [ 804.318550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.375388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 804.412703] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 805.404518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.429434] LNet: 43874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.436697] LNetError: 43874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.456134] LNet: Removed LNI 192.168.201.48@tcp [ 805.540971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.659073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 805.700569] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 812.747037] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:37:36 (1790692656) [ 812.992450] LNet: 44465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.005617] LNetError: 44465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.041677] LNet: Removed LNI 192.168.201.48@tcp [ 813.764374] Key type .llcrypt unregistered [ 813.765405] Key type ._llcrypt unregistered [ 820.690263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.907126] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.924505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.928954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.567954] Key type ._llcrypt registered [ 823.570622] Key type .llcrypt registered [ 823.786381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.807886] alg: No test for adler32 (adler32-zlib) [ 824.734521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.809711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.851966] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 825.556939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 825.573772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.419761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.435269] LNet: 45202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.443851] LNetError: 45202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 827.448535] LNet: Removed LNI 10.1.2.3@tcp [ 827.458980] LNet: Removed LNI 192.168.201.48@tcp [ 827.476768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.549475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 827.571555] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 827.574976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.179081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 828.202983] LNet: 45229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.208940] LNetError: 45229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 828.214953] LNetError: 45229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 828.221676] LNet: Removed LNI 10.1.2.3@tcp [ 828.238457] LNet: Removed LNI 192.168.201.48@tcp [ 828.267103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.349429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 828.374451] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 828.379846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.734665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 828.755933] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 829.485330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.501263] LNet: 45392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.508041] LNetError: 45392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 829.515438] LNetError: 45392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 829.529682] LNet: Removed LNI 10.1.2.3@tcp [ 829.538066] LNet: Removed LNI 192.168.201.48@tcp [ 829.552326] LNet: Removed LNI 192.168.201.48@tcp2 [ 829.566094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.607573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 829.621192] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 829.623800] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.627332] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 835.999845] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:38:00 (1790692680) [ 836.154045] LNet: 45983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.161611] LNetError: 45983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.174761] LNetError: 45983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 836.199153] LNet: Removed LNI 10.1.2.3@tcp [ 836.208615] LNet: Removed LNI 192.168.201.48@tcp [ 836.245112] LNet: Removed LNI 192.168.201.48@tcp2 [ 836.904315] Key type .llcrypt unregistered [ 836.906398] Key type ._llcrypt unregistered [ 843.904164] Key type ._llcrypt registered [ 843.910463] Key type .llcrypt registered [ 844.159856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.190993] alg: No test for adler32 (adler32-zlib) [ 845.209681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.277726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.323677] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 846.317853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.335682] LNet: 46382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.346656] LNetError: 46382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.359574] LNet: Removed LNI 192.168.201.48@tcp [ 846.396402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.523308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.550341] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 847.361315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.374485] LNet: 46412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.379705] LNetError: 46412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.388974] LNet: Removed LNI 192.168.201.48@tcp [ 847.404244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.450388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.471388] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 848.163666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.194249] LNet: Removed LNI 192.168.201.48@tcp [ 848.214093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.279399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.307072] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 848.929196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.144112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.151244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.160227] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.627241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.651787] LNet: 46519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.658340] LNet: 46519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 851.670317] LNetError: 46519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.674417] LNetError: 46519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.707179] LNet: Removed LNI 192.168.201.48@tcp [ 851.741652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.873585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.910985] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 859.378603] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:38:23 (1790692703) [ 859.510879] LNet: 47108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.516787] LNetError: 47108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.539363] LNet: Removed LNI 192.168.201.48@tcp [ 860.269216] Key type .llcrypt unregistered [ 860.274322] Key type ._llcrypt unregistered [ 867.707063] Key type ._llcrypt registered [ 867.709507] Key type .llcrypt registered [ 868.018043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.044715] alg: No test for adler32 (adler32-zlib) [ 869.023507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.089716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.110687] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 869.861354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.879572] LNet: 47513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.885488] LNetError: 47513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.897696] LNet: Removed LNI 192.168.201.48@tcp [ 869.911123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.968682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 869.991902] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 876.640442] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:38:40 (1790692720) [ 876.773861] LNet: 48099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.784101] LNetError: 48099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.821429] LNet: Removed LNI 192.168.201.48@tcp [ 877.489885] Key type .llcrypt unregistered [ 877.491773] Key type ._llcrypt unregistered [ 884.078530] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.235744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 884.239658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 884.244573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.931534] Key type ._llcrypt registered [ 886.934578] Key type .llcrypt registered [ 887.194295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.224894] alg: No test for adler32 (adler32-zlib) [ 888.162992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.216787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.240729] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 889.024915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.041651] LNet: 48556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.046530] LNetError: 48556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.068269] LNet: Removed LNI 192.168.201.48@tcp [ 889.111034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.165584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 889.185476] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 895.635273] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:38:59 (1790692739) [ 895.759943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.771808] LNet: 49148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.777638] LNetError: 49148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.794780] LNet: Removed LNI 192.168.201.48@tcp [ 895.838491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.993246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 902.225396] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:39:06 (1790692746) [ 902.465601] LNet: 49735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.237680] Key type .llcrypt unregistered [ 903.240109] Key type ._llcrypt unregistered [ 911.722594] Key type ._llcrypt registered [ 911.724404] Key type .llcrypt registered [ 911.875534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.890469] alg: No test for adler32 (adler32-zlib) [ 912.992423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.082512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.126238] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 914.493070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.511299] LNet: 50136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.520720] LNetError: 50136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.550444] LNet: Removed LNI 192.168.201.48@tcp [ 914.572226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.648825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 914.667032] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 924.012637] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:39:27 (1790692767) [ 924.148788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.180747] LNet: 50723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.187773] LNetError: 50723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.212114] LNet: Removed LNI 192.168.201.48@tcp [ 924.267297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.419147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 931.566663] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:39:35 (1790692775) [ 933.378478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 934.880634] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:39:39 (1790692779) [ 935.939238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 937.999905] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:39:41 (1790692781) [ 939.659505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 940.932437] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:39:45 (1790692785) [ 942.823848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 944.142931] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:39:48 (1790692788) [ 945.676780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 947.370688] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:39:51 (1790692791) [ 947.474824] LNet: 52745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.035447] Key type .llcrypt unregistered [ 948.039261] Key type ._llcrypt unregistered [ 953.629863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.711092] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.715036] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.717830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.099694] Key type ._llcrypt registered [ 956.100678] Key type .llcrypt registered [ 956.308705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.332358] alg: No test for adler32 (adler32-zlib) [ 957.240590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.310678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.335795] LNet: Added LNI 192.168.201.48@tcp [8/512/0/180] [ 957.347531] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 957.349836] LNet: 53073:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.463617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.485524] LNet: 53092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.494686] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.503931] LNet: Removed LNI 10.1.2.3@tcp [ 957.510132] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.536793] LNet: Removed LNI 192.168.201.48@tcp [ 957.556804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.632145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.653897] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 957.724843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.750711] LNet: 53116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.756917] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.767837] LNet: Removed LNI 192.168.201.48@tcp [ 957.793457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.852153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 957.871813] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 957.878471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.881372] LNet: 53124:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 957.953389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.969145] LNet: 53142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.979807] LNetError: 53142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.987626] LNet: Removed LNI 10.1.2.3@tcp [ 957.994731] LNetError: 53142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.010656] LNet: Removed LNI 192.168.201.48@tcp [ 958.023695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.095294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.121292] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 958.128481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 958.131095] LNet: 53150:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 958.293660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.312276] LNet: 53176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.315485] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.318455] LNet: Removed LNI 10.1.2.3@tcp [ 958.320106] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.329338] LNet: Removed LNI 192.168.201.48@tcp [ 958.349361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.416635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.434879] LNet: Added LNI 192.168.201.48@tcp [8/256/0/182] [ 958.443526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 958.572557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.590457] LNet: 53204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.595649] LNetError: 53204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.603703] LNet: Removed LNI 10.1.2.3@tcp [ 958.606082] LNetError: 53204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.622053] LNet: Removed LNI 192.168.201.48@tcp [ 958.653947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.734454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 958.772867] LNet: Added LNI 192.168.201.48@tcp [8/256/0/182] [ 958.778209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 958.781477] LNet: 53212:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 958.852306] LNet: Removed LNI 10.1.2.3@tcp [ 958.861771] LNet: Removed LNI 192.168.201.48@tcp [ 959.455312] Key type .llcrypt unregistered [ 959.459575] Key type ._llcrypt unregistered [ 971.168810] Key type ._llcrypt registered [ 971.183394] Key type .llcrypt registered [ 972.657162] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:40:16 (1790692816) [ 973.736165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 975.409518] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:40:19 (1790692819) [ 975.928504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 975.937389] alg: No test for adler32 (adler32-zlib) [ 977.020683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.104454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.132724] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 978.373377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.48 [ 978.404354] LNet: 54475:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 978.413473] LNet: Removed LNI 192.168.201.48@tcp [ 985.879300] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:40:29 (1790692829) [ 986.417806] LNet: 55045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.431665] LNetError: 55045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.468258] LNet: Removed LNI 192.168.201.48@tcp [ 987.294519] Key type .llcrypt unregistered [ 987.296782] Key type ._llcrypt unregistered [ 994.165026] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.431461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.915038] Key type ._llcrypt registered [ 996.916430] Key type .llcrypt registered [ 997.105192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.126504] alg: No test for adler32 (adler32-zlib) [ 998.666386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.921555] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:40:49 (1790692849) [ 1005.992704] LNet: 55948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.002437] LNetError: 55948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.034189] LNet: Removed LNI 10.1.2.3@tcp [ 1006.874206] Key type .llcrypt unregistered [ 1006.876645] Key type ._llcrypt unregistered [ 1013.401872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.716818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1013.721957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1013.730804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1015.230372] Key type ._llcrypt registered [ 1015.231539] Key type .llcrypt registered [ 1015.347156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.357469] alg: No test for adler32 (adler32-zlib) [ 1016.363129] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.335353] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:41:07 (1790692867) [ 1024.803836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1026.486088] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:41:10 (1790692870) [ 1026.746073] LNet: 57132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.768602] LNetError: 57132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1026.796455] LNet: Removed LNI 10.1.2.3@tcp [ 1027.538499] Key type .llcrypt unregistered [ 1027.540479] Key type ._llcrypt unregistered [ 1035.520807] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1035.830141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1035.844726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1035.852513] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1038.597882] Key type ._llcrypt registered [ 1038.600852] Key type .llcrypt registered [ 1038.861294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.897261] alg: No test for adler32 (adler32-zlib) [ 1039.899789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.966871] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.399315] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:41:32 (1790692892) [ 1048.895542] LNet: 58362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.906346] LNetError: 58362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.918301] LNet: Removed LNI 10.1.2.3@tcp [ 1049.588900] Key type .llcrypt unregistered [ 1049.591743] Key type ._llcrypt unregistered [ 1056.910542] Key type ._llcrypt registered [ 1056.914254] Key type .llcrypt registered [ 1057.112931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.151878] alg: No test for adler32 (adler32-zlib) [ 1058.184069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1058.219834] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1066.205283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1066.845371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_interrupt -r 1 [ 1066.903322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_interrupt -r 1 [ 1066.957530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1066.988661] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1067.280643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.313736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.338966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.359463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.403452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1067.448259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.488571] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1067.498742] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1067.512569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.548365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.579995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.595763] LNet: 58800:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.615251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.676338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.702863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.727302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.753504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.779145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.807210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.240776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_dropped -r 1 [ 1068.303919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_dropped -r 1 [ 1068.354947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1068.738709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.760611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.806147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.845734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.887728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1068.925978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.962397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.011260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.070634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.102207] LNet: 58912:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.132478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.170247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.197380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.232825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.260818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.292979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.332234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.825658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_aborted -r 1 [ 1069.853670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_aborted -r 1 [ 1069.878497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1069.892411] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1069.896342] LNet: Skipped 1 previous similar message [ 1070.091709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.123731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1070.157529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.189892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.213187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1070.232879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.260410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.285191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.311200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.324154] LNet: 59023:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.338286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.372429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.406497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.443630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.473749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.501321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.532470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1071.063166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_no_route -r 1 [ 1071.109079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_no_route -r 1 [ 1071.141248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1071.385319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.435345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.481606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.521232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.579231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1071.612924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.653784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.697558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.745188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.808977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.855546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.903748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.945665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.007782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.067653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.153542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1072.709187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_timeout -r 1 [ 1072.747967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_timeout -r 1 [ 1072.805461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1072.827412] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1072.831189] LNet: Skipped 1 previous similar message [ 1073.135541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.167304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.197030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.221346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.256746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1073.285508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.315239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.344734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.373447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.394265] LNet: 59245:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1073.401868] LNet: 59245:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1073.413965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.460613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.501632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.533720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.560687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.591809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.641791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1074.234804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_error -r 1 [ 1074.274343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_error -r 1 [ 1074.311255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1074.608248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.664208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.703419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.738224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.787524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1074.814758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.851541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.889288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.929272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.956644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.978953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.008494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.039488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.090642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.140423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.172689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1075.431973] LNet: 59398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.436916] LNetError: 59398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.464864] LNet: Removed LNI 192.168.201.48@tcp [ 1076.990775] Key type .llcrypt unregistered [ 1076.992838] Key type ._llcrypt unregistered [ 1094.071325] Key type ._llcrypt registered [ 1094.073193] Key type .llcrypt registered [ 1095.691792] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:42:19 (1790692939) [ 1096.486313] Key type .llcrypt unregistered [ 1096.488080] Key type ._llcrypt unregistered [ 1103.226360] Key type ._llcrypt registered [ 1103.228330] Key type .llcrypt registered [ 1103.431949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1103.452299] alg: No test for adler32 (adler32-zlib) [ 1104.431170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1104.465957] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1112.685621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1123.591339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1123.653880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1123.689605] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1124.694757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1125.471600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_interrupt -r 1 [ 1125.512456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_interrupt -r 1 [ 1125.558037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_interrupt -r 1 [ 1125.627777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_interrupt -r 1 [ 1125.674187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_interrupt -r 1 [ 1125.713254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_interrupt -r 1 [ 1125.756665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_interrupt -r 1 [ 1125.797952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_interrupt -r 1 [ 1125.827901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1125.843504] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1125.850248] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp1 [ 1126.126299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.160342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.191478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.228715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.253233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1126.283790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.317941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.345771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.378415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.398816] LNet: 60874:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.423674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.472818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.508824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.535700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.563713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.592436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.623427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.141749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_dropped -r 1 [ 1127.171611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_dropped -r 1 [ 1127.211313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_dropped -r 1 [ 1127.245693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_dropped -r 1 [ 1127.272402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_dropped -r 1 [ 1127.291345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_dropped -r 1 [ 1127.311880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_dropped -r 1 [ 1127.345776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_dropped -r 1 [ 1127.411728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1127.449968] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1127.453033] LNet: Skipped 2 previous similar messages [ 1127.719619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.738196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1127.754317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.774554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.792723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1127.820823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.849883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.884600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.923680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.943259] LNet: 60997:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.963669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.988515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.014416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.044847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.071801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.098569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.136639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.636877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_aborted -r 1 [ 1128.709453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_aborted -r 1 [ 1128.823672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_aborted -r 1 [ 1128.884559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_aborted -r 1 [ 1128.921355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_aborted -r 1 [ 1128.964790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_aborted -r 1 [ 1129.008448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_aborted -r 1 [ 1129.039563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_aborted -r 1 [ 1129.097671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1129.114182] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1129.119269] LNet: Skipped 2 previous similar messages [ 1129.341783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.370470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.405362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.435140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.464367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1129.491991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.526878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.558139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.587698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.602497] LNet: 61121:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.619674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.644039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.680626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.718864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.744389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.795640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.823973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.192609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_no_route -r 1 [ 1130.216234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_no_route -r 1 [ 1130.243251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_no_route -r 1 [ 1130.269148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_no_route -r 1 [ 1130.303461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_no_route -r 1 [ 1130.331204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_no_route -r 1 [ 1130.359908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_no_route -r 1 [ 1130.386669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_no_route -r 1 [ 1130.413795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1130.619969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.646866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.683422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.701634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.723627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1130.744624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.765804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.792407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.815443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.837763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.868634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.920468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.948399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.972764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.999028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.035606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1131.613702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_timeout -r 1 [ 1131.670680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_timeout -r 1 [ 1131.708368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_timeout -r 1 [ 1131.759984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_timeout -r 1 [ 1131.813538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_timeout -r 1 [ 1131.869271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_timeout -r 1 [ 1131.897694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_timeout -r 1 [ 1131.938464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_timeout -r 1 [ 1131.997373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1132.030053] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1132.039638] LNet: Skipped 5 previous similar messages [ 1132.314583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.348444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.380415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.417744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.447418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1132.496534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.538444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.577402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.637375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.668164] LNet: 61367:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.678564] LNet: 61367:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1132.709945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.754439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.783317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.811293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.844233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.871560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.898380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1133.463541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_error -r 1 [ 1133.507314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_error -r 1 [ 1133.535200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_error -r 1 [ 1133.564033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_error -r 1 [ 1133.636975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_error -r 1 [ 1133.671692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_error -r 1 [ 1133.698206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_error -r 1 [ 1133.742121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_error -r 1 [ 1133.787890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1134.153765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.193827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.235434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.279060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.322522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1134.360291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.418313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.461633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.522393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.565210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.634606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.689830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.736246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.778758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.829265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.865554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.021482] LNet: 61567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.038147] LNetError: 61567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.046421] LNet: Removed LNI 192.168.201.48@tcp [ 1140.057977] LNet: Removed LNI 192.168.201.48@tcp1 [ 1140.569754] Key type .llcrypt unregistered [ 1140.571684] Key type ._llcrypt unregistered [ 1158.820237] Key type ._llcrypt registered [ 1158.822760] Key type .llcrypt registered [ 1160.277602] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:43:24 (1790693004) [ 1161.495319] Key type .llcrypt unregistered [ 1161.498856] Key type ._llcrypt unregistered [ 1168.777283] Key type ._llcrypt registered [ 1168.780566] Key type .llcrypt registered [ 1169.017071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1169.038197] alg: No test for adler32 (adler32-zlib) [ 1170.031769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1170.049480] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1178.352910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1179.121136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_dropped -r 1 [ 1179.179663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_dropped -r 1 [ 1179.209493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1179.242664] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1179.629733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.667629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.700653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.758745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.843966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.911794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.969087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.020645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.048262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.067984] LNet: 62820:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.093740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.125483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.164488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.204916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.241476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.285682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.315681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.881986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_error -r 1 [ 1180.912923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_error -r 1 [ 1180.950153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1180.965416] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1181.300114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.344727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.376483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.409336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.434950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1181.463305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.522407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.579444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.640163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.682819] LNet: 62931:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.713227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.757526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.789417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.833486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.883817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.927579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.971179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.574883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_timeout -r 1 [ 1182.623717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_timeout -r 1 [ 1182.650142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1182.668901] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1182.937527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.973306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.019405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.063546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.111191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1183.158381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.221519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.281409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.315342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.336565] LNet: 63042:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.357533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.409428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.453967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.501898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.543911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.615462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.659566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.057906] LNet: 63083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.064909] LNetError: 63083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.133417] LNet: Removed LNI 192.168.201.48@tcp [ 1185.697507] Key type .llcrypt unregistered [ 1185.699203] Key type ._llcrypt unregistered [ 1202.071854] Key type ._llcrypt registered [ 1202.074344] Key type .llcrypt registered [ 1203.389243] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:44:07 (1790693047) [ 1204.165468] Key type .llcrypt unregistered [ 1204.172081] Key type ._llcrypt unregistered [ 1209.685714] Key type ._llcrypt registered [ 1209.686848] Key type .llcrypt registered [ 1209.975752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1210.000464] alg: No test for adler32 (adler32-zlib) [ 1211.019995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1211.066550] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1219.306630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1230.571749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.606809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1230.621310] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1231.594458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1232.044769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_dropped -r 1 [ 1232.068389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e remote_dropped -r 1 [ 1232.094643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_dropped -r 1 [ 1232.121292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e remote_dropped -r 1 [ 1232.152662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e remote_dropped -r 1 [ 1232.185381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e remote_dropped -r 1 [ 1232.213867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e remote_dropped -r 1 [ 1232.241102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e remote_dropped -r 1 [ 1232.279802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1232.303969] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1232.307577] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.148@tcp1 [ 1232.485254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.512303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.536553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.558339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.586683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1232.614752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.640325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.665339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.688409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.701797] LNet: 64558:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.715892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.746858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.780876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.800787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.825815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.847975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.877325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.901718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.308990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_error -r 1 [ 1233.342864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e remote_error -r 1 [ 1233.411935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_error -r 1 [ 1233.432950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e remote_error -r 1 [ 1233.457409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e remote_error -r 1 [ 1233.494562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e remote_error -r 1 [ 1233.542469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e remote_error -r 1 [ 1233.616169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e remote_error -r 1 [ 1233.707598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1233.745578] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1233.759665] LNet: Skipped 2 previous similar messages [ 1234.130602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.187326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.215258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.253780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.302536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1234.353759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.405983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.449235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.520982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.557808] LNet: 64683:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.578565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.626661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.653678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.711307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.763949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.821280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.876971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1234.926508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.644726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_timeout -r 1 [ 1235.723767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e remote_timeout -r 1 [ 1235.803827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_timeout -r 1 [ 1235.852419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e remote_timeout -r 1 [ 1235.911755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e remote_timeout -r 1 [ 1235.964184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e remote_timeout -r 1 [ 1236.021865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e remote_timeout -r 1 [ 1236.065878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e remote_timeout -r 1 [ 1236.149883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.148@tcp [ 1236.159802] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1236.163576] LNet: Skipped 1 previous similar message [ 1236.402329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.437641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.475458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.498670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.519874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1236.536800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.555451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.574639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.595558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.605776] LNet: 64809:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.620799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.675430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.692503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.718089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.742060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.763033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.775637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.790067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.348715] LNet: 64887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.354230] LNetError: 64887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.363835] LNet: Removed LNI 192.168.201.48@tcp [ 1241.370911] LNet: Removed LNI 192.168.201.48@tcp1 [ 1241.972512] Key type .llcrypt unregistered [ 1241.980096] Key type ._llcrypt unregistered [ 1260.871771] Key type ._llcrypt registered [ 1260.878107] Key type .llcrypt registered [ 1262.328245] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:45:06 (1790693106) [ 1263.940686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1265.599537] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:45:09 (1790693109) [ 1266.324322] Key type .llcrypt unregistered [ 1266.329713] Key type ._llcrypt unregistered [ 1272.687949] Key type ._llcrypt registered [ 1272.689971] Key type .llcrypt registered [ 1272.973068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.994339] alg: No test for adler32 (adler32-zlib) [ 1273.993203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1274.044898] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1282.936846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1283.656696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e network_timeout -r 1 [ 1283.699558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e network_timeout -r 1 [ 1283.780874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1283.804518] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1284.029160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.061774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.116798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.161730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.205633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1284.240297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.269630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.312797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.320658] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1284.321942] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1284.358996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.402303] LNet: 66432:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1284.440713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.477047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.514450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.553787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.588561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.618938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.969778] LNet: 66471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.983141] LNetError: 66471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.009296] LNet: Removed LNI 192.168.201.48@tcp [ 1285.407718] Key type .llcrypt unregistered [ 1285.410463] Key type ._llcrypt unregistered [ 1305.887325] Key type ._llcrypt registered [ 1305.889325] Key type .llcrypt registered [ 1306.132612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1306.147387] alg: No test for adler32 (adler32-zlib) [ 1307.099190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1307.146719] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1315.465405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1325.264713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.327709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.367894] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1326.190795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1326.828370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e network_timeout -r 1 [ 1326.860510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e network_timeout -r 1 [ 1326.889392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e network_timeout -r 1 [ 1326.938655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e network_timeout -r 1 [ 1326.992022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e network_timeout -r 1 [ 1327.045636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e network_timeout -r 1 [ 1327.076662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e network_timeout -r 1 [ 1327.124115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e network_timeout -r 1 [ 1327.181643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1327.212384] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1327.477242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1327.514275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1327.557495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1327.594659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1327.625387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1327.655332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.661529] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp1 [ 1327.676573] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.148@tcp1 [ 1327.700196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.735616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.790356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1327.808674] LNet: 67359:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1327.852321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.894149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.923322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.946143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.996871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1328.037823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.038757] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1333.531771] LNet: 67433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.537196] LNetError: 67433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.561713] LNet: Removed LNI 192.168.201.48@tcp [ 1334.566443] LNetError: 67433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1334.584515] LNet: Removed LNI 192.168.201.48@tcp1 [ 1335.047504] Key type .llcrypt unregistered [ 1335.049516] Key type ._llcrypt unregistered [ 1349.091657] Key type ._llcrypt registered [ 1349.093742] Key type .llcrypt registered [ 1350.481330] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:46:34 (1790693194) [ 1351.007290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.033743] alg: No test for adler32 (adler32-zlib) [ 1352.167185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.254170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1352.282023] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1353.290920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1353.314881] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1354.501870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 1354.589239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1354.689503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1354.750516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1354.861757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1354.917838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1354.960340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.48@tcp [ 1354.987974] LNetError: 68540:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.48@tcp to recovery [ 1355.296271] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1355.296895] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1357.353936] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1357.746983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.444545] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1362.621817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1362.648426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.686593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.697312] LNet: 68647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.700779] LNetError: 68647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.744233] LNet: Removed LNI 192.168.201.48@tcp [ 1363.747087] LNetError: 68647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1363.777954] LNet: Removed LNI 192.168.201.48@tcp1 [ 1363.820897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.879608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.905296] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1365.307316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1365.335145] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1366.286529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 1366.335670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.421581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1366.488431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1366.552706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1366.623633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1366.652762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.679879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.48@tcp [ 1366.699876] LNetError: 68920:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.48@tcp to recovery [ 1366.944343] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1369.307834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1372.064750] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1372.069591] LNet: Skipped 1 previous similar message [ 1376.415827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1376.459899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.496311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1383.266935] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:47:07 (1790693227) [ 1383.513605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1383.542596] LNet: 69617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.550019] LNetError: 69617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.558973] LNet: Removed LNI 192.168.201.48@tcp [ 1383.570895] LNet: Removed LNI 192.168.201.48@tcp1 [ 1383.605905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.679990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1383.725047] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1384.512513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.535608] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1385.356323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 1385.438437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1385.464311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.502435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.530980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1385.559478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.597166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.48@tcp --health 0 [ 1385.697376] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.48@tcp [ 1385.697677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1386.720406] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1386.725902] LNet: Skipped 1 previous similar message [ 1400.169727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1400.389854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1400.433750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1400.465433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.48@tcp --health 500 [ 1400.624896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.403844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.466732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.486624] LNet: 70093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.490728] LNetError: 70093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.494232] LNetError: 70093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1407.526059] LNet: Removed LNI 192.168.201.48@tcp [ 1407.553248] LNet: Removed LNI 192.168.201.48@tcp1 [ 1407.604601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.724765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1407.771894] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1408.575738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.600800] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1409.741226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 1409.817823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.939386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1410.016454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1410.074448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1410.146407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1410.210842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1410.297287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.48@tcp --health 0 [ 1411.808547] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1411.812080] LNet: Skipped 3 previous similar messages [ 1412.957450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.561091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1421.604431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1421.644467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1428.348287] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:47:52 (1790693272) [ 1428.830387] LNet: 71129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.835253] LNetError: 71129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.849431] LNetError: 71129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1428.855484] LNet: Removed LNI 192.168.201.48@tcp [ 1428.870309] LNet: Removed LNI 192.168.201.48@tcp1 [ 1429.483470] Key type .llcrypt unregistered [ 1429.486055] Key type ._llcrypt unregistered [ 1436.442228] Key type ._llcrypt registered [ 1436.444152] Key type .llcrypt registered [ 1436.760293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1436.794618] alg: No test for adler32 (adler32-zlib) [ 1437.775308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1437.798284] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1447.603856] Lustre: DEBUG MARKER: Initial discovery [ 1447.676874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.148@tcp [ 1450.191613] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1450.287834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1450.320571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.148@tcp [ 1450.346941] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1450.373593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1450.439799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.148@tcp -r 1 -m GET -l 3 [ 1450.482918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.148@tcp -r 1 -m GET -e local_error [ 1450.513809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.148@tcp -r 1 -m PUT -l 6 [ 1451.104383] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1451.749804] Lustre: DEBUG MARKER: Force oleg148-client.virtnet to discover 192.168.201.148@tcp (in background) [ 1451.820022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.148@tcp [ 1453.503392] Lustre: DEBUG MARKER: Force oleg148-server to discover 192.168.201.48@tcp [ 1454.884594] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1455.901405] Lustre: DEBUG MARKER: Wait for 71884 [ 1457.304280] Lustre: DEBUG MARKER: Finished wait on 71884 [ 1457.447759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1457.497089] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1457.518772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.792465] LNet: 72341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.799220] LNetError: 72341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.831804] LNet: Removed LNI 192.168.201.48@tcp [ 1458.329616] Key type .llcrypt unregistered [ 1458.332264] Key type ._llcrypt unregistered [ 1475.593230] Key type ._llcrypt registered [ 1475.594667] Key type .llcrypt registered [ 1476.918196] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:48:40 (1790693320) [ 1477.401480] Key type .llcrypt unregistered [ 1477.406357] Key type ._llcrypt unregistered [ 1485.498819] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1485.716894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1485.723246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1485.732690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1488.211511] Key type ._llcrypt registered [ 1488.219322] Key type .llcrypt registered [ 1488.491409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1488.527508] alg: No test for adler32 (adler32-zlib) [ 1489.486344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1489.552562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1489.579704] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1490.322356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1490.344370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1498.976529] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:49:03 (1790693343) [ 1499.247786] LNet: 74420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.255871] LNetError: 74420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1499.275742] LNet: Removed LNI 10.1.2.3@tcp [ 1499.301494] LNet: Removed LNI 192.168.201.48@tcp [ 1499.893283] Key type .llcrypt unregistered [ 1499.895995] Key type ._llcrypt unregistered [ 1507.180619] Key type ._llcrypt registered [ 1507.182853] Key type .llcrypt registered [ 1507.444687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.462232] alg: No test for adler32 (adler32-zlib) [ 1508.445598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.500493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.516488] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1509.431413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1509.477877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1510.698713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.48@tcp [ 1511.776884] LNetError: Unexpected error -22 connecting to 192.168.201.48@tcp at host 192.168.201.48:988 [ 1517.022687] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:49:21 (1790693361) [ 1517.176845] LNet: 75547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.184049] LNetError: 75547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1517.195805] LNet: Removed LNI 10.1.2.3@tcp [ 1517.207374] LNet: Removed LNI 192.168.201.48@tcp [ 1518.092754] Key type .llcrypt unregistered [ 1518.096771] Key type ._llcrypt unregistered [ 1525.086370] Key type ._llcrypt registered [ 1525.089542] Key type .llcrypt registered [ 1525.348455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1525.368618] alg: No test for adler32 (adler32-zlib) [ 1526.267862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.307567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1526.329908] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1527.017863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1527.038759] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 1527.824775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.48@tcp1 --nid 192.168.201.48@tcp2 [ 1532.673546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp1 192.168.201.48@tcp2 [ 1532.692376] LNetError: 76388:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.48@tcp1 to 192.168.201.48@tcp2 - Route table may be misconfigured [ 1532.720194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp2 192.168.201.48@tcp1 [ 1539.981463] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:49:43 (1790693383) [ 1540.149935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.182098] LNet: 76960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.198737] LNetError: 76960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.214064] LNet: Removed LNI 192.168.201.48@tcp1 [ 1541.289420] LNet: Removed LNI 192.168.201.48@tcp2 [ 1541.375950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.453108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.502290] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1542.445444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1542.503821] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1543.327565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 1543.386428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.48@tcp [ 1543.422330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp1 192.168.201.48@tcp1 [ 1543.467462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.48@tcp -d 192.168.201.48@tcp -e network_timeout [ 1543.531914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -e network_timeout [ 1543.570296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -e network_timeout [ 1543.604577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -e network_timeout [ 1543.646308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1543.674483] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1543.696510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1543.793260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1543.855561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1544.480239] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1550.754523] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:49:54 (1790693394) [ 1550.854877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.897134] LNet: 77819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.905215] LNetError: 77819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.913421] LNetError: 77819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1551.968172] LNet: Removed LNI 192.168.201.48@tcp [ 1551.970332] LNetError: 77819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.007286] LNet: Removed LNI 192.168.201.48@tcp1 [ 1552.063594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.177437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1552.532603] LNet: 77861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.813676] Key type .llcrypt unregistered [ 1552.814790] Key type ._llcrypt unregistered [ 1565.133630] Key type ._llcrypt registered [ 1565.135402] Key type .llcrypt registered [ 1566.717953] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:50:10 (1790693410) [ 1567.154372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1567.172827] alg: No test for adler32 (adler32-zlib) [ 1568.108159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.553581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1571.088792] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:50:15 (1790693415) [ 1571.357355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.376240] LNet: 78960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.398337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.463667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1571.483479] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1572.092508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1572.131390] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1572.919929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1572.992068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp1 [ 1573.049715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp1 [ 1579.107643] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:50:23 (1790693423) [ 1581.076199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg148-server [ 1582.613229] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:50:26 (1790693426) [ 1584.543626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg148-server [ 1586.379272] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:50:30 (1790693430) [ 1588.266820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg148-server [ 1590.262538] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:50:34 (1790693434) [ 1591.839642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg148-server [ 1593.685671] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:50:37 (1790693437) [ 1595.176633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg148-server [ 1596.614670] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:50:41 (1790693441) [ 1598.171933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg148-server [ 1599.658337] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:50:44 (1790693444) [ 1602.012532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg148-server [ 1603.742094] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:50:47 (1790693447) [ 1606.366400] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg148-server [ 1608.434899] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:50:52 (1790693452) [ 1611.270554] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg148-server [ 1612.852974] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:50:57 (1790693457) [ 1613.109708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.120965] LNet: 82837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.126051] LNetError: 82837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.134862] LNet: Removed LNI 192.168.201.48@tcp [ 1614.183638] LNet: Removed LNI 192.168.201.48@tcp1 [ 1614.234407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.321842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.355931] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1615.521297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1615.608176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1615.834473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.878795] LNet: 82985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.888728] LNetError: 82985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1615.896835] LNetError: 82985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1615.909067] LNet: Removed LNI 192.168.201.48@tcp [ 1615.930713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.006346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1616.031129] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1616.835799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1617.037925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.068474] LNetError: 83130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.090332] LNet: Removed LNI 192.168.201.48@tcp [ 1618.148650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.230845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.248964] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1619.034505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1619.145345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1620.410846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.428956] LNet: 83287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.433936] LNet: 83287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1620.444292] LNetError: 83287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.481192] LNet: Removed LNI 192.168.201.48@tcp [ 1621.520992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.654413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1621.719989] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1622.436362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1622.611639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.656771] LNet: Removed LNI 192.168.201.48@tcp [ 1623.746271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.869600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.903913] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1624.991022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1625.119359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1625.412369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.465624] LNet: 83579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.488054] LNet: 83579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1625.505710] LNetError: 83579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1625.520390] LNetError: 83579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1625.562967] LNet: Removed LNI 192.168.201.48@tcp [ 1625.624501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.729531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1625.764122] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1626.719970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1626.950648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.025062] LNet: Removed LNI 192.168.201.48@tcp [ 1628.055901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.142857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.169333] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1628.957782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1629.039393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1629.226382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.311046] LNet: Removed LNI 192.168.201.48@tcp [ 1630.355732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.458221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1630.522445] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1631.303991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1631.496930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1632.553169] LNet: Removed LNI 192.168.201.48@tcp [ 1632.594242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.708455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.744721] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1633.545158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1633.647784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1633.992789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.013788] LNet: 84165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.018661] LNet: 84165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1634.027238] LNetError: 84165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.032491] LNetError: 84165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1635.050046] LNet: Removed LNI 192.168.201.48@tcp [ 1635.112286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.215392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1635.267917] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1636.276789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1636.464641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.543052] LNet: Removed LNI 192.168.201.48@tcp [ 1637.586270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.654811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.692575] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1638.674478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1638.786546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1639.099716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.169366] LNet: Removed LNI 192.168.201.48@tcp [ 1640.228254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.349776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1640.424935] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1641.205550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1643.937324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.030910] LNet: Removed LNI 192.168.201.48@tcp [ 1645.095830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.158516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.181388] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1645.876603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1645.953507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.996933] LNet: Removed LNI 192.168.201.48@tcp [ 1646.021884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.106874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.157873] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1646.776608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1646.920887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.005314] LNet: Removed LNI 192.168.201.48@tcp [ 1647.037326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.120320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1647.219724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.258398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.346669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1647.441755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.483640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.544843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1647.630597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.662147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.752847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1647.785796] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1648.508539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.531792] LNet: Removed LNI 192.168.201.48@tcp [ 1648.561848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.660825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1655.651647] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:51:39 (1790693499) [ 1655.975840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.012644] LNet: 85643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.015558] LNet: 85643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1656.036204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.105554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1656.143259] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/180] [ 1657.567697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.610336] LNetError: 85781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.613166] LNetError: 85781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1657.630278] LNet: Removed LNI 192.168.201.48@tcp231 [ 1657.658260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.757834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1657.817737] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/180] [ 1657.827632] LNet: 85789:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1658.483182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1658.519456] LNet: Removed LNI 192.168.201.48@tcp231 [ 1658.531533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1658.565572] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/180] [ 1660.279290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.306165] LNet: Removed LNI 192.168.201.48@tcp231 [ 1660.324909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.382162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1660.403746] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/0] [ 1660.408564] LNet: 85941:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1660.915226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1660.962149] LNet: Removed LNI 192.168.201.48@tcp231 [ 1660.990140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1661.051494] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/0] [ 1662.702792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.762304] LNet: Removed LNI 192.168.201.48@tcp231 [ 1662.825067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.954757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1663.011772] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/60] [ 1663.017492] LNet: 86093:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1663.668640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1663.706922] LNet: Removed LNI 192.168.201.48@tcp231 [ 1663.722345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1663.747504] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/60] [ 1664.942941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.986067] LNet: Removed LNI 192.168.201.48@tcp231 [ 1665.034749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.123412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1665.152819] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/180] [ 1665.160293] LNet: 86244:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1665.811150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1665.875783] LNet: Removed LNI 192.168.201.48@tcp231 [ 1665.903587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1665.927897] LNet: Added LNI 192.168.201.48@tcp231 [8/256/0/180] [ 1674.742925] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:51:58 (1790693518) [ 1674.986826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.020568] LNet: Removed LNI 192.168.201.48@tcp231 [ 1675.041519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.108988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1675.131597] LNet: Added LNI 192.168.201.48@tcp232 [8/256/0/180] [ 1675.938643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1676.014711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1676.052165] LNet: Removed LNI 192.168.201.48@tcp232 [ 1676.063668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1676.086925] LNet: Added LNI 192.168.201.48@tcp232 [8/256/0/180] [ 1676.884437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1676.924978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1676.952671] LNet: Removed LNI 192.168.201.48@tcp232 [ 1676.973576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1677.007472] LNet: Added LNI 192.168.201.48@tcp232 [8/256/0/180] [ 1677.861171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1677.885726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1677.905613] LNet: Removed LNI 192.168.201.48@tcp232 [ 1677.916093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1677.935519] LNet: Added LNI 192.168.201.48@tcp232 [8/256/0/180] [ 1678.620528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1678.652403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1678.691865] LNet: Removed LNI 192.168.201.48@tcp232 [ 1678.707710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1685.006122] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:52:09 (1790693529) [ 1686.155987] Key type .llcrypt unregistered [ 1686.157896] Key type ._llcrypt unregistered [ 1693.519419] Key type ._llcrypt registered [ 1693.523738] Key type .llcrypt registered [ 1693.822987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1693.866152] alg: No test for adler32 (adler32-zlib) [ 1694.825873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1694.870927] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1702.606291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1712.146565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.209654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1712.246500] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1713.251494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1713.893814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_interrupt -r 2 [ 1713.931774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_interrupt -r 2 [ 1713.967985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_interrupt -r 2 [ 1714.000072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_interrupt -r 2 [ 1714.038763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_interrupt -r 2 [ 1714.068698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_interrupt -r 2 [ 1714.102927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_interrupt -r 2 [ 1714.137888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_interrupt -r 2 [ 1714.185502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1714.208601] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1714.213673] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1714.486890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1714.514895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1714.539166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1714.564828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1714.596236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1714.621658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1714.650896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1714.690295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1714.730363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1714.747460] LNet: 88750:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1714.786165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1714.843819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1714.906265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1714.959828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1714.995173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1715.043842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1715.713914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_dropped -r 2 [ 1715.766518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_dropped -r 2 [ 1715.817805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_dropped -r 2 [ 1715.912778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_dropped -r 2 [ 1715.986440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_dropped -r 2 [ 1716.024392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_dropped -r 2 [ 1716.070426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_dropped -r 2 [ 1716.105738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_dropped -r 2 [ 1716.157383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1716.180853] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1716.448848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1716.489701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1716.525778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1716.569924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1716.605453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1716.639572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1716.666274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1716.698293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1716.726914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1716.743890] LNet: 88873:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.762958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1716.795574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1716.821560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1716.851914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1716.884893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1716.922996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1717.479646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_aborted -r 2 [ 1717.562796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_aborted -r 2 [ 1717.631164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_aborted -r 2 [ 1717.687359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_aborted -r 2 [ 1717.750942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_aborted -r 2 [ 1717.812786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_aborted -r 2 [ 1717.872757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_aborted -r 2 [ 1717.920571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_aborted -r 2 [ 1717.951492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1717.966912] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1718.224966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1718.271373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1718.308995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1718.346997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1718.402891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1718.447438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1718.478171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1718.521710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1718.557914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1718.598221] LNet: 88995:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.613893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1718.636494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1718.659431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1718.684541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1718.708571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1718.735392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1719.400806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_no_route -r 2 [ 1719.447208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_no_route -r 2 [ 1719.482340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_no_route -r 2 [ 1719.513815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_no_route -r 2 [ 1719.575760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_no_route -r 2 [ 1719.624800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_no_route -r 2 [ 1719.691869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_no_route -r 2 [ 1719.761560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_no_route -r 2 [ 1719.845196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1720.223496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1720.255814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1720.301213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1720.340891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1720.371211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1720.417172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1720.484216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1720.556681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1720.626424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1720.658947] LNet: 89117:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1720.691430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1720.738378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1720.775484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1720.840405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1720.871459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1720.937905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1721.487970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e local_timeout -r 2 [ 1721.568160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e local_timeout -r 2 [ 1721.629216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e local_timeout -r 2 [ 1721.713683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e local_timeout -r 2 [ 1721.773697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e local_timeout -r 2 [ 1721.829988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e local_timeout -r 2 [ 1721.869852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e local_timeout -r 2 [ 1721.911006] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e local_timeout -r 2 [ 1721.954654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1721.973929] LNet: There was an unexpected network error while writing to 192.168.201.148: rc = -22 [ 1721.983680] LNet: Skipped 1 previous similar message [ 1722.368204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1722.405421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1722.439082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1722.483772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1722.532769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1722.578770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1722.632502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1722.683907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1722.745241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1722.780447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1722.813205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1722.845771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1722.863372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1722.886682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1722.902479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1723.567427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp -m GET -e remote_dropped -r 2 [ 1723.607762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.148@tcp1 -m GET -e remote_dropped -r 2 [ 1723.659776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp -m GET -e remote_dropped -r 2 [ 1723.712199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp -d 192.168.201.48@tcp1 -m GET -e remote_dropped -r 2 [ 1723.755971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp -m GET -e remote_dropped -r 2 [ 1723.819093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.148@tcp1 -m GET -e remote_dropped -r 2 [ 1723.872527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp -m GET -e remote_dropped -r 2 [ 1723.922221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.48@tcp1 -d 192.168.201.48@tcp1 -m GET -e remote_dropped -r 2 [ 1723.988563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.48@tcp 192.168.201.148@tcp [ 1724.013800] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.148@tcp [ 1724.301617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1724.341377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1724.367666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1724.398939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1724.442740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1724.467744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1724.495153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1724.545954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1724.600441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1724.627577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1724.662907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1724.690837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1724.718281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1724.748496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1724.799803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1731.052188] LNet: 89436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.062339] LNetError: 89436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.066215] LNet: Removed LNI 192.168.201.48@tcp [ 1731.080576] LNet: Removed LNI 192.168.201.48@tcp1 [ 1731.467246] Key type .llcrypt unregistered [ 1731.469639] Key type ._llcrypt unregistered [ 1749.467231] Key type ._llcrypt registered [ 1749.469292] Key type .llcrypt registered [ 1750.733618] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:53:14 (1790693594) [ 1751.194933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1751.225594] alg: No test for adler32 (adler32-zlib) [ 1752.135505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.177765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.197304] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1752.947500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1752.974203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.48@tcp [ 1753.014620] LNetError: 90414:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.48@tcp to recovery [ 1753.094597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.48@tcp [ 1753.238664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1761.517965] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:53:25 (1790693605) [ 1761.844413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.862952] LNet: 91003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.867895] LNetError: 91003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.897067] LNet: Removed LNI 192.168.201.48@tcp [ 1761.929662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.971095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.993217] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1762.484820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1762.497510] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 1763.430689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.48@tcp [ 1763.465142] LNetError: 91270:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.48@tcp to recovery [ 1764.000909] LNet: 1 local NIs in recovery (showing 1): 192.168.201.48@tcp [ 1764.006590] LNet: There was an unexpected network error while writing to 192.168.201.48: rc = -22 [ 1764.762802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.48@tcp [ 1774.461796] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:53:38 (1790693618) [ 1777.336708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg148-server [ 1778.901511] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:53:43 (1790693623) [ 1781.960449] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg148-server [ 1784.809829] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:53:47 (1790693627) [ 1787.619446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg148-server [ 1789.616480] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:53:53 (1790693633) [ 1789.885458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.924573] LNet: 92901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.941160] LNetError: 92901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.944368] LNet: Removed LNI 192.168.201.48@tcp [ 1789.969516] LNet: Removed LNI 192.168.201.48@tcp1 [ 1790.043639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.129543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.159640] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1790.995150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1791.071600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1791.136430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1791.235657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1791.286166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1791.359639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1797.662372] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:54:02 (1790693642) [ 1798.211217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.252478] LNet: 93632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.280493] LNetError: 93632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.294551] LNetError: 93632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1798.317078] LNet: Removed LNI 192.168.201.48@tcp [ 1798.377646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.446584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.463242] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1806.715759] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:54:10 (1790693650) [ 1806.874711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.959358] LNet: 94334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.973661] LNetError: 94334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.994783] LNet: Removed LNI 192.168.201.48@tcp [ 1807.024967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.093747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.112872] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1814.788204] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:54:18 (1790693658) [ 1816.743814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1819.194459] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:54:22 (1790693662) [ 1819.947651] LNet: 95371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.954565] LNetError: 95371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.984975] LNet: Removed LNI 192.168.201.48@tcp [ 1821.071662] Key type .llcrypt unregistered [ 1821.073456] Key type ._llcrypt unregistered [ 1830.434187] Key type ._llcrypt registered [ 1830.436787] Key type .llcrypt registered [ 1830.885468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1830.929685] alg: No test for adler32 (adler32-zlib) [ 1831.956446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1832.021472] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1840.825593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 1847.261954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.148@tcp [ 1847.657525] LNet: 95789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.664086] LNetError: 95789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.673756] LNet: Removed LNI 192.168.201.48@tcp [ 1848.029364] Key type .llcrypt unregistered [ 1848.031210] Key type ._llcrypt unregistered [ 1860.385523] Key type ._llcrypt registered [ 1860.388725] Key type .llcrypt registered [ 1861.835920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1863.615856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1865.226758] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:55:09 (1790693709) [ 1866.685331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1868.478150] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:55:12 (1790693712) [ 1870.332434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg148-server [ 1872.297193] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:55:16 (1790693716) [ 1874.399990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg148-server [ 1876.176001] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:55:20 (1790693720) [ 1878.051552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1878.138441] alg: No test for adler32 (adler32-zlib) [ 1879.236429] Lustre: Lustre: Build Version: 2.17.59_2_g85450b5 [ 1879.622168] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1881.312971] Key type lgssc registered [ 1882.536314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1903.438176] Key type lgssc unregistered [ 1903.804760] LNet: 98972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.816279] LNetError: 98972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.835907] LNet: Removed LNI 192.168.201.48@tcp [ 1904.648683] Key type .llcrypt unregistered [ 1904.656360] Key type ._llcrypt unregistered [ 1920.029848] Key type ._llcrypt registered [ 1920.033938] Key type .llcrypt registered [ 1921.465943] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:56:05 (1790693765) [ 1922.068894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.085727] alg: No test for adler32 (adler32-zlib) [ 1923.210893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.381377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1923.480252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1923.566381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1923.614959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1932.134383] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:56:15 (1790693775) [ 1932.247519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.289197] LNet: 100390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.317802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.384367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.431924] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1933.689287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.705576] LNet: 100541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.708986] LNetError: 100541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.723161] LNet: Removed LNI 192.168.201.48@tcp [ 1933.741439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.802691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1933.839841] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1934.627075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.670974] LNetError: 100569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1934.705723] LNet: Removed LNI 192.168.201.48@tcp [ 1934.725650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.865499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1934.918569] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1943.648151] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:56:27 (1790693787) [ 1943.762165] LNet: 101151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.766209] LNet: 101151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1943.774222] LNetError: 101151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.792630] LNet: Removed LNI 192.168.201.48@tcp [ 1944.410716] Key type .llcrypt unregistered [ 1944.413796] Key type ._llcrypt unregistered [ 1952.335396] Key type ._llcrypt registered [ 1952.342305] Key type .llcrypt registered [ 1952.558789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.568432] alg: No test for adler32 (adler32-zlib) [ 1953.668715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.803309] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 1953.805408] LNet: *** cfs_fail_loc=e003, val=100*** [ 1954.916593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 1954.966324] LNet: *** cfs_fail_loc=e003, val=100*** [ 1954.971601] LNet: Skipped 2 previous similar messages [ 1954.985720] Ping reply received with mapping: [ 1954.991087] found metadata reply NI[192.168.201.48@tcp] [ 1954.996153] 30 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1955.017830] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1955.019776] 5a 5a 5a 5a 5a 5a 5a 5a [ 1955.354728] LNet: 101556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.375319] LNetError: 101556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.391341] LNet: Removed LNI 192.168.201.48@tcp [ 1957.479874] Key type .llcrypt unregistered [ 1957.481797] Key type ._llcrypt unregistered [ 1971.341612] Key type ._llcrypt registered [ 1971.343942] Key type .llcrypt registered [ 1972.617198] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:56:56 (1790693816) [ 1972.990819] Key type .llcrypt unregistered [ 1972.996047] Key type ._llcrypt unregistered [ 1979.597683] Key type ._llcrypt registered [ 1979.599527] Key type .llcrypt registered [ 1979.819501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.838555] alg: No test for adler32 (adler32-zlib) [ 1980.904700] LNetError: 102598:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1981.240625] Key type .llcrypt unregistered [ 1981.244171] Key type ._llcrypt unregistered [ 1987.055198] Key type ._llcrypt registered [ 1987.056798] Key type .llcrypt registered [ 1988.523176] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:57:12 (1790693832) [ 1989.042972] Key type .llcrypt unregistered [ 1989.047174] Key type ._llcrypt unregistered [ 1997.465317] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1997.470732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2000.191794] Key type ._llcrypt registered [ 2000.196788] Key type .llcrypt registered [ 2000.503801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2000.531525] alg: No test for adler32 (adler32-zlib) [ 2001.476469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.525722] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2012.104553] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:57:36 (1790693856) [ 2012.200306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.255520] LNet: 104371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.264488] LNetError: 104371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2012.290972] LNet: Removed LNI 10.1.2.3@tcp [ 2012.319245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.234923] Key type .llcrypt unregistered [ 2013.238326] Key type ._llcrypt unregistered [ 2020.610452] Key type ._llcrypt registered [ 2020.614853] Key type .llcrypt registered [ 2020.950562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.971873] alg: No test for adler32 (adler32-zlib) [ 2021.927574] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2022.000670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2022.175413] LNet: 104665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.181641] LNetError: 104665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.210686] LNet: Removed LNI 192.168.201.48@tcp [ 2022.913041] Key type .llcrypt unregistered [ 2022.914223] Key type ._llcrypt unregistered [ 2034.318084] Key type ._llcrypt registered [ 2034.321551] Key type .llcrypt registered [ 2035.751693] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:57:59 (1790693879) [ 2036.368838] Key type .llcrypt unregistered [ 2036.373491] Key type ._llcrypt unregistered [ 2043.206426] Key type ._llcrypt registered [ 2043.211454] Key type .llcrypt registered [ 2043.432113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2043.459654] alg: No test for adler32 (adler32-zlib) [ 2050.965256] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:58:15 (1790693895) [ 2051.376959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.441115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.506438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2051.540766] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2052.069557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.094456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2052.116166] LNet: Removed LNI 192.168.201.48@tcp [ 2052.401635] LNet: 106409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.813457] Key type .llcrypt unregistered [ 2052.815810] Key type ._llcrypt unregistered [ 2066.134696] Key type ._llcrypt registered [ 2066.138210] Key type .llcrypt registered [ 2067.677208] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:58:31 (1790693911) [ 2068.194913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.203683] alg: No test for adler32 (adler32-zlib) [ 2069.228521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.281660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.303655] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2070.249646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2070.300948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.48@tcp [ 2070.337921] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2070.350050] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2070.356924] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2070, 0, 0 [ 2070.366589] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2070.370447] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2070.376961] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2070.384137] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2070, 0, 0 [ 2070.395036] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2070.400801] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2070.412332] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2070.426255] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2070, 0, 2120 [ 2070.443768] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2070.456059] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2070.457618] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2070.459543] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2070, 0, 2120 [ 2070.472621] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2078.104177] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:58:41 (1790693921) [ 2078.660442] LNet: 107948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.673594] LNetError: 107948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.721828] LNet: Removed LNI 192.168.201.48@tcp [ 2080.654176] Key type .llcrypt unregistered [ 2080.662460] Key type ._llcrypt unregistered [ 2088.474795] Key type ._llcrypt registered [ 2088.484399] Key type .llcrypt registered [ 2088.785402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.804722] alg: No test for adler32 (adler32-zlib) [ 2089.786615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2089.828683] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2098.324887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 2107.628645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.671725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2107.700990] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 2108.541627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 2108.680791] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2108.823631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2108.825776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2108.838361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2111.214645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2111.250983] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2122.305650] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:59:26 (1790693966) [ 2122.679413] LNet: 109458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.685919] LNetError: 109458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.744230] LNet: Removed LNI 192.168.201.48@tcp [ 2123.752402] LNetError: 109458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.767576] LNet: Removed LNI 192.168.201.48@tcp1 [ 2123.809125] LNet: Removed LNI 10.1.2.3@tcp99 [ 2124.617652] Key type .llcrypt unregistered [ 2124.620924] Key type ._llcrypt unregistered [ 2130.637209] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2130.640949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2133.036738] Key type ._llcrypt registered [ 2133.038548] Key type .llcrypt registered [ 2133.126733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2133.136979] alg: No test for adler32 (adler32-zlib) [ 2134.045698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.098683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.123340] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2134.541788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2134.569507] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2136.300837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2136.341988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2136.386444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2136.403735] LNetError: 110201:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2136.434969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2136.477790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2136.874485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2136.932513] LNet: 110221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.937889] LNetError: 110221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2136.942574] LNet: Removed LNI 10.1.2.3@tcp [ 2137.964769] LNet: Removed LNI 192.168.201.48@tcp [ 2138.593748] Key type .llcrypt unregistered [ 2138.595228] Key type ._llcrypt unregistered [ 2155.022170] Key type ._llcrypt registered [ 2155.025480] Key type .llcrypt registered [ 2156.489849] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:00:00 (1790694000) [ 2156.898993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2156.913659] alg: No test for adler32 (adler32-zlib) [ 2157.897945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.975926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.997497] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2158.775139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2158.927717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg148-client@tcp [ 2165.895179] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:00:09 (1790694009) [ 2166.111021] LNet: 111753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.119573] LNetError: 111753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.177631] LNet: Removed LNI 192.168.201.48@tcp [ 2166.824403] Key type .llcrypt unregistered [ 2166.833430] Key type ._llcrypt unregistered [ 2174.179693] Key type ._llcrypt registered [ 2174.185315] Key type .llcrypt registered [ 2174.373291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2174.403862] alg: No test for adler32 (adler32-zlib) [ 2175.387514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2175.403492] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2175.427532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2182.628028] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:00:26 (1790694026) [ 2184.311634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2186.093579] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:00:30 (1790694030) [ 2187.667938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2189.381967] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:00:33 (1790694033) [ 2189.680173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.714509] LNet: 113180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.725614] LNetError: 113180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.767647] LNet: Removed LNI 192.168.201.48@tcp [ 2189.813961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.879122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.905695] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2190.777277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2190.844490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2197.641680] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:00:41 (1790694041) [ 2197.747914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.772840] LNet: 113884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.780786] LNetError: 113884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.812902] LNet: Removed LNI 192.168.201.48@tcp [ 2197.858146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.916777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2197.946367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2198.662687] Key type .llcrypt unregistered [ 2198.669670] Key type ._llcrypt unregistered [ 2209.424715] Key type ._llcrypt registered [ 2209.426024] Key type .llcrypt registered [ 2210.715813] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:00:54 (1790694054) [ 2211.304589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.334510] alg: No test for adler32 (adler32-zlib) [ 2212.337078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.386509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.430695] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2213.118239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2213.186535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2219.449339] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:01:03 (1790694063) [ 2219.647123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.686171] LNet: 115454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.693809] LNetError: 115454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.708235] LNet: Removed LNI 192.168.201.48@tcp [ 2219.770524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.839202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2219.891446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2227.149208] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:01:11 (1790694071) [ 2227.250805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.279967] LNet: 116030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.327654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.426863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2227.819204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2227.869926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2227.962929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2228.315137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2228.345474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2228.426982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.454296] LNet: 116070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.473057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.573375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2228.971866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2229.017345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2229.092547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2229.388336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2229.414800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2235.298429] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:01:19 (1790694079) [ 2235.394087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.436780] LNet: 116667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.472983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.554987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.582124] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2236.166287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2236.184454] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2236.972541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2237.022924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2237.219475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.238294] LNetError: 116950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.304163] LNet: Removed LNI 192.168.201.48@tcp [ 2238.313185] LNetError: 116950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.336479] LNet: Removed LNI 192.168.201.48@tcp2 [ 2238.359716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.444720] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2238.451541] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2238.457230] LNet: 116956:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.075997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2239.317300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.352905] LNet: Removed LNI 192.168.201.48@tcp [ 2240.356467] LNetError: 116993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.362949] LNetError: 116993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2240.374829] LNet: Removed LNI 192.168.201.48@tcp2 [ 2240.418724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.515905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.552906] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2241.477849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2241.493616] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2242.218365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2242.253854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2242.327946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2242.670745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2242.700579] LNet: 117278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.709643] LNet: 117278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2243.744182] LNet: Removed LNI 192.168.201.48@tcp [ 2244.776768] LNet: Removed LNI 192.168.201.48@tcp2 [ 2244.845905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.930799] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2244.939603] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2244.944326] LNet: 117284:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.639970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2245.991704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.018349] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.023512] LNetError: 117321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2246.029259] LNet: Removed LNI 192.168.201.48@tcp [ 2247.078798] LNet: Removed LNI 192.168.201.48@tcp2 [ 2247.133567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.233884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2247.267841] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2248.061247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2248.086545] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2249.196673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2249.283764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2249.509537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.595083] LNet: Removed LNI 192.168.201.48@tcp [ 2250.615753] LNet: Removed LNI 192.168.201.48@tcp2 [ 2250.645611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.726261] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2250.734890] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2250.740189] LNet: 117607:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.495995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2251.709629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.725491] LNet: 117645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.731454] LNet: 117645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2252.769300] LNet: Removed LNI 192.168.201.48@tcp [ 2252.789736] LNet: Removed LNI 192.168.201.48@tcp2 [ 2252.856893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.979198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2252.997237] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2253.795073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2253.825203] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2254.806888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2254.841989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2254.918231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2255.132524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.151476] LNetError: 117929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.160933] LNetError: 117929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2255.166545] LNet: Removed LNI 192.168.201.48@tcp [ 2256.233166] LNet: Removed LNI 192.168.201.48@tcp2 [ 2256.281771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.430073] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2256.442210] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2256.453954] LNet: 117936:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2257.129347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2257.290670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.311896] LNet: Removed LNI 192.168.201.48@tcp [ 2258.346211] LNet: Removed LNI 192.168.201.48@tcp2 [ 2258.405961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.487660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.528209] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2259.237412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2259.284708] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2260.359771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2260.387836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2260.454464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2260.944460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.984722] LNet: Removed LNI 192.168.201.48@tcp [ 2261.997930] LNet: Removed LNI 192.168.201.48@tcp2 [ 2262.058877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.147533] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2262.152089] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2262.155943] LNet: 118277:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2262.782522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2263.265788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.353101] LNet: Removed LNI 192.168.201.48@tcp [ 2264.371544] LNet: Removed LNI 192.168.201.48@tcp2 [ 2264.405501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.508026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.538978] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2265.368204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2265.435266] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2266.653754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2266.697551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2266.729539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2266.777853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2267.168477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.192760] LNet: Removed LNI 192.168.201.48@tcp [ 2269.227487] LNet: Removed LNI 192.168.201.48@tcp2 [ 2269.261544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.319662] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2269.324796] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2270.189816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2276.805583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2278.283243] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:02:02 (1790694122) [ 2278.425259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.460805] LNet: 119391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.476378] LNet: 119391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2278.487222] LNetError: 119391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.495426] LNetError: 119391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2278.506824] LNet: Removed LNI 192.168.201.48@tcp [ 2279.527407] LNet: Removed LNI 192.168.201.48@tcp2 [ 2279.593674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.968783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2280.000068] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2280.814382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2280.828363] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2281.680807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.707649] LNet: Removed LNI 10.1.2.3@tcp [ 2281.721739] LNet: Removed LNI 192.168.201.48@tcp [ 2281.743556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.792987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2281.810648] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2281.815651] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2282.165578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2282.204405] LNet: Removed LNI 192.168.201.48@tcp [ 2282.216333] LNet: Removed LNI 10.1.2.3@tcp [ 2288.939819] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:02:12 (1790694132) [ 2289.113125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.216279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.389335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.432411] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2290.601517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2290.639249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2290.735074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2290.802799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2290.867468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2290.926472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2290.988926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.028489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.071428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.138479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.216311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.279233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2291.522197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2298.925494] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 11:02:23 (1790694143) [ 2298.987357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.081280] LNet: Removed LNI 192.168.201.48@tcp [ 2300.118900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.229478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2300.251048] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2301.178612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2301.210896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2301.269590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.321783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.352450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.388860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.433232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.488361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.523285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.552586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.595510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.640630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.728762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2301.761967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.794570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.822658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.866848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.907602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.932356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2301.968714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2302.005458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2302.035713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2302.065758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.48@tcp [ 2308.882757] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:02:32 (1790694152) [ 2309.277703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.377972] LNet: Removed LNI 192.168.201.48@tcp [ 2310.427933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.639113] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2310.913173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2310.922839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2310.935290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2313.302678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.335227] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2314.321781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2314.339160] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2316.601855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2316.637945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2316.702054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.48@tcp -m PUT -e local_timeout -r 1 [ 2316.747925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2316.808183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2316.812704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2317.057470] LNetError: 121835:0:(lib-move.c:4794:lnet_parse()) 192.168.201.48@tcp, src 192.168.201.48@tcp: Dropping PUT (error -108 looking up sender) [ 2318.118834] LNet: 122293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.148215] LNet: 122293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2318.178676] LNetError: 122293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2318.186620] LNetError: 122293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2318.210376] LNet: Removed LNI 10.1.2.3@tcp [ 2319.293109] LNet: Removed LNI 192.168.201.48@tcp [ 2319.816309] Key type .llcrypt unregistered [ 2319.818392] Key type ._llcrypt unregistered [ 2333.647333] Key type ._llcrypt registered [ 2333.648610] Key type .llcrypt registered [ 2334.926772] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:02:58 (1790694178) [ 2335.491449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.514967] alg: No test for adler32 (adler32-zlib) [ 2336.536987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.976514] LNet: 123146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.639557] Key type .llcrypt unregistered [ 2337.642430] Key type ._llcrypt unregistered [ 2344.899858] Key type ._llcrypt registered [ 2344.902265] Key type .llcrypt registered [ 2345.108148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2345.137274] alg: No test for adler32 (adler32-zlib) [ 2346.093947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2346.113725] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2354.091688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 2354.348287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2355.201407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.148@tcp [ 2355.361825] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.148@tcp [ 2355.430892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.456668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.482932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.506504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.529276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2355.561333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.589968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.630542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2355.673524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.696969] LNet: 123598:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.713355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.757727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.781358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.808343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.839766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.878978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.527594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2356.761077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.796347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2356.831155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.854968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.881469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2356.917049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.948431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.975928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.008965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.024683] LNet: 123717:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.039279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.067378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.093520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.124673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.151431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.179486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.713090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.148@tcp [ 2357.999257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2358.050401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2358.079613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2358.125908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2358.161867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2358.193957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2358.225184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.279492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.315246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.339912] LNet: 123837:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.367954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.419457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.462360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.513492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.589614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.636198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2359.188398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2359.465723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2359.493852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2359.523446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2359.545873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2359.574087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2359.595552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2359.621461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2359.642477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2359.665794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2359.688277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2359.716573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2359.741749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2359.764984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2359.795561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2359.825576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2360.156291] LNet: 124002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2360.166240] LNetError: 124002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.190057] LNet: Removed LNI 192.168.201.48@tcp [ 2360.623508] Key type .llcrypt unregistered [ 2360.631179] Key type ._llcrypt unregistered [ 2378.037457] Key type ._llcrypt registered [ 2378.039591] Key type .llcrypt registered [ 2379.161379] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:03:43 (1790694223) [ 2379.644982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.657346] alg: No test for adler32 (adler32-zlib) [ 2380.608766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.965372] LNet: 124889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.650348] Key type .llcrypt unregistered [ 2381.651934] Key type ._llcrypt unregistered [ 2388.654378] Key type ._llcrypt registered [ 2388.659935] Key type .llcrypt registered [ 2388.905869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2388.927851] alg: No test for adler32 (adler32-zlib) [ 2389.886151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2389.915429] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2398.119899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 2409.377046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.445230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2409.497197] LNet: Added LNI 192.168.201.48@tcp1 [8/256/0/180] [ 2410.503961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.148@tcp [ 2410.656400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2411.596946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.148@tcp [ 2411.942561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2411.973910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2412.018326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2412.069093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2412.127378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2412.164365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2412.192500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2412.224782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2412.259465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2412.282672] LNet: 125548:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2412.309426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2412.345331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2412.389328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2412.461451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2412.528994] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.148@tcp [ 2412.544061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2412.590702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2413.564768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2413.970124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2414.041240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2414.112678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2414.172490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2414.241598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.293923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.337132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.381119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.424862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.452038] LNet: 125668:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.495781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.555269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.641570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.711890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.761911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.825828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.751672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.148@tcp [ 2416.128932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.179316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.227723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.270410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.310371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2416.352919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.408952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.467605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.567422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.608526] LNet: 125787:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.630623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.678352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.720210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.769821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.805538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.843928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.707442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2418.024613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.061973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.089312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.114885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.146111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2418.216906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.267245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.331697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.370653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.428366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.483251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.532688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.593790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.686849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.742406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.751414] LNet: 125988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.765155] LNetError: 125988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.773520] LNet: Removed LNI 192.168.201.48@tcp [ 2424.785298] LNet: Removed LNI 192.168.201.48@tcp1 [ 2425.341839] Key type .llcrypt unregistered [ 2425.344212] Key type ._llcrypt unregistered [ 2443.137233] Key type ._llcrypt registered [ 2443.139646] Key type .llcrypt registered [ 2444.488603] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:04:48 (1790694288) [ 2444.997778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2445.013464] alg: No test for adler32 (adler32-zlib) [ 2445.905787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.950829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.983440] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2447.089050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2447.130281] LNet: Added LNI 192.168.201.48@tcp2 [8/256/0/180] [ 2448.180339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.48@tcp [ 2448.281531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2449.314745] LNet: Removed LNI 192.168.201.48@tcp [ 2456.219497] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:05:00 (1790694300) [ 2458.018901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2459.770830] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:05:03 (1790694303) [ 2467.073714] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:05:11 (1790694311) [ 2475.646975] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:05:19 (1790694319) [ 2483.841277] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:05:27 (1790694327) [ 2491.649038] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:05:35 (1790694335) [ 2499.918612] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:05:43 (1790694343) [ 2507.859883] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:05:51 (1790694351) [ 2516.124334] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:05:59 (1790694359) [ 2526.271707] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:06:09 (1790694369) [ 2533.733972] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:06:17 (1790694377) [ 2541.774574] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:06:25 (1790694385) [ 2549.129223] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:06:33 (1790694393) [ 2556.709729] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:06:40 (1790694400) [ 2563.615677] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:06:47 (1790694407) [ 2572.246376] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:06:56 (1790694416) [ 2581.037831] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:07:04 (1790694424) [ 2590.628447] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:07:14 (1790694434) [ 2599.669668] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:07:23 (1790694443) [ 2599.956280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2600.013910] LNet: 137683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.029984] LNetError: 137683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.058826] LNet: Removed LNI 192.168.201.48@tcp2 [ 2600.085118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2600.207383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2600.226254] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2601.821183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2601.843492] LNet: 137827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2601.850708] LNetError: 137827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2601.869916] LNet: Removed LNI 192.168.201.48@tcp [ 2601.905560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2601.981504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2602.007597] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2609.565857] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:07:33 (1790694453) [ 2617.734332] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:07:41 (1790694461) [ 2627.722302] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:07:51 (1790694471) [ 2635.762480] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:08:00 (1790694480) [ 2643.868429] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:08:07 (1790694487) [ 2652.921530] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:08:16 (1790694496) [ 2661.425822] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:08:25 (1790694505) [ 2669.502697] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:08:33 (1790694513) [ 2677.675857] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:08:41 (1790694521) [ 2685.318462] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:08:49 (1790694529) [ 2692.455908] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:08:56 (1790694536) [ 2700.218764] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:09:03 (1790694543) [ 2708.458252] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2494 sec ========== 11:09:12 (1790694552) [ 2708.872385] LNet: 145296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.881609] LNetError: 145296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.898168] LNet: Removed LNI 192.168.201.48@tcp [ 2709.616657] Key type .llcrypt unregistered [ 2709.621809] Key type ._llcrypt unregistered [ 2723.089355] Key type ._llcrypt registered [ 2723.095547] Key type .llcrypt registered [ 2723.625538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2723.641553] alg: No test for adler32 (adler32-zlib) [ 2725.025897] Lustre: Lustre: Build Version: 2.17.59_2_g85450b5 [ 2725.348816] LNet: Added LNI 192.168.201.48@tcp [8/256/0/180] [ 2727.088723] Key type lgssc registered [ 2728.545057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2801.355588] Lustre: Mounted lustre-client - version 2.17.59_2_g85450b5 [ 2807.936669] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2815.627132] Lustre: Unmounted lustre-client [ 2839.689619] Key type lgssc unregistered [ 2839.919224] LNet: 148267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2839.925331] LNetError: 148267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2839.946083] LNet: Removed LNI 192.168.201.48@tcp [ 2840.779270] Key type .llcrypt unregistered [ 2840.782288] Key type ._llcrypt unregistered