[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 418194097 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002317] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007551] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.012021] LSM: Security Framework initializing [ 0.013059] Yama: becoming mindful. [ 0.014039] SELinux: Initializing. [ 0.015075] *** VALIDATE selinux *** [ 0.023186] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028326] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030095] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031111] *** VALIDATE tmpfs *** [ 0.032444] *** VALIDATE proc *** [ 0.034027] *** VALIDATE cgroup *** [ 0.034954] *** VALIDATE cgroup2 *** [ 0.036218] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037138] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039029] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042693] debug: unmapping init [mem 0xffffffff94c59000-0xffffffff94c60fff] [ 0.044868] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045575] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046022] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049009] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053253] rcu: Hierarchical SRCU implementation. [ 0.055468] smp: Bringing up secondary CPUs ... [ 0.056571] x86: Booting SMP configuration: [ 0.057024] .... node #0, CPUs: #1 #2 #3 [ 0.060464] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.127283] node 0 deferred pages initialised in 62ms [ 0.129090] devtmpfs: initialized [ 0.130396] x86/mm: Memory block size: 128MB [ 0.133395] gcov: version magic: 0x41383552 [ 0.135218] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.136108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.137244] pinctrl core: initialized pinctrl subsystem [ 0.138196] [ 0.138686] ************************************************************* [ 0.139011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.140012] ** ** [ 0.141032] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142019] ** ** [ 0.143014] ** This means that this kernel is built to expose internal ** [ 0.144014] ** IOMMU data structures, which may compromise security on ** [ 0.145020] ** your system. ** [ 0.146014] ** ** [ 0.147019] ** If you see this message and you are not debugging the ** [ 0.148014] ** kernel, report this immediately to your vendor! ** [ 0.149013] ** ** [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151032] ************************************************************* [ 0.152789] NET: Registered protocol family 16 [ 0.153348] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.154073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156566] cpuidle: using governor menu [ 0.158754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.161487] PCI: Using configuration type 1 for base access [ 0.163178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.174036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.176053] cryptd: max_cpu_qlen set to 1000 [ 0.178090] ACPI: Added _OSI(Module Device) [ 0.179013] ACPI: Added _OSI(Processor Device) [ 0.180013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.181015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.185159] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.189587] ACPI: Interpreter enabled [ 0.190070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.191015] ACPI: Using IOAPIC for interrupt routing [ 0.192124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.193412] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.203403] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.209019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.213071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218452] acpiphp: Slot [2] registered [ 0.220127] acpiphp: Slot [5] registered [ 0.222143] acpiphp: Slot [6] registered [ 0.224125] acpiphp: Slot [3] registered [ 0.225098] acpiphp: Slot [4] registered [ 0.227107] acpiphp: Slot [7] registered [ 0.229162] acpiphp: Slot [8] registered [ 0.230105] acpiphp: Slot [9] registered [ 0.232123] acpiphp: Slot [10] registered [ 0.234100] acpiphp: Slot [11] registered [ 0.235178] acpiphp: Slot [12] registered [ 0.237113] acpiphp: Slot [13] registered [ 0.238096] acpiphp: Slot [14] registered [ 0.240155] acpiphp: Slot [15] registered [ 0.241085] acpiphp: Slot [16] registered [ 0.242090] acpiphp: Slot [17] registered [ 0.244144] acpiphp: Slot [18] registered [ 0.246096] acpiphp: Slot [19] registered [ 0.247334] acpiphp: Slot [20] registered [ 0.249092] acpiphp: Slot [21] registered [ 0.251110] acpiphp: Slot [22] registered [ 0.252083] acpiphp: Slot [23] registered [ 0.254124] acpiphp: Slot [24] registered [ 0.255101] acpiphp: Slot [25] registered [ 0.257103] acpiphp: Slot [26] registered [ 0.259186] acpiphp: Slot [27] registered [ 0.261106] acpiphp: Slot [28] registered [ 0.262103] acpiphp: Slot [29] registered [ 0.264089] acpiphp: Slot [30] registered [ 0.265117] acpiphp: Slot [31] registered [ 0.266048] PCI host bridge to bus 0000:00 [ 0.268024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.271024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.283174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.286898] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290426] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.297749] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.301010] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.311577] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315990] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319101] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321649] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.326013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.338017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.342883] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.348501] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.353013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.358013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.375026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.382734] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.425000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.427387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.430349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.433397] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.435172] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.439136] iommu: Default domain type: Passthrough [ 0.441453] SCSI subsystem initialized [ 0.443132] ACPI: bus type USB registered [ 0.445188] usbcore: registered new interface driver usbfs [ 0.448089] usbcore: registered new interface driver hub [ 0.450062] usbcore: registered new device driver usb [ 0.452201] pps_core: LinuxPPS API ver. 1 registered [ 0.454012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.457066] PTP clock support registered [ 0.459108] EDAC MC: Ver: 3.0.0 [ 0.461124] PCI: Using ACPI for IRQ routing [ 0.462541] NetLabel: Initializing [ 0.464013] NetLabel: domain hash size = 128 [ 0.465008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.467081] NetLabel: unlabeled traffic allowed by default [ 0.469128] vgaarb: loaded [ 0.470276] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.472011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.478004] clocksource: Switched to clocksource kvm-clock [ 0.583071] VFS: Disk quotas dquot_6.6.0 [ 0.584456] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.586986] *** VALIDATE ramfs *** [ 0.588176] *** VALIDATE hugetlbfs *** [ 0.590349] pnp: PnP ACPI init [ 0.592722] pnp: PnP ACPI: found 6 devices [ 0.617912] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.622144] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.625024] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.627540] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.630418] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.633083] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.635756] NET: Registered protocol family 2 [ 0.638476] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.642630] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.646571] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.651041] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.654489] TCP: Hash tables configured (established 65536 bind 65536) [ 0.657451] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.660299] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.662933] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.665954] NET: Registered protocol family 1 [ 0.668794] RPC: Registered named UNIX socket transport module. [ 0.671391] RPC: Registered udp transport module. [ 0.673309] RPC: Registered tcp transport module. [ 0.675536] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.678050] NET: Registered protocol family 44 [ 0.679113] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.680337] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.681618] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.682994] PCI: CLS 0 bytes, default 64 [ 0.684606] Unpacking initramfs... [ 2.091112] debug: unmapping init [mem 0xffff8c6dfcc64000-0xffff8c6dfffcffff] [ 2.096456] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.098356] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.100750] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.597121] Initialise system trusted keyrings [ 2.598589] Key type blacklist registered [ 2.600526] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.611583] zbud: loaded [ 2.614839] *** VALIDATE nfs *** [ 2.615970] *** VALIDATE nfs4 *** [ 2.617655] pstore: using deflate compression [ 2.620585] Platform Keyring initialized [ 2.724982] NET: Registered protocol family 38 [ 2.727308] Key type asymmetric registered [ 2.728608] Asymmetric key parser 'x509' registered [ 2.730762] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.734560] io scheduler mq-deadline registered [ 2.736061] io scheduler kyber registered [ 2.737398] io scheduler bfq registered [ 2.739102] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.741302] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.743539] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.745878] ACPI: Power Button [PWRF] [ 2.751344] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.759285] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.773279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.801490] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.830768] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.836843] Non-volatile memory driver v1.3 [ 2.839081] Linux agpgart interface v0.103 [ 2.872556] virtio_blk virtio1: [vda] 146696 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.875692] vda: detected capacity change from 0 to 75108352 [ 2.891135] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.894200] vdb: detected capacity change from 0 to 1073741824 [ 2.903576] libphy: Fixed MDIO Bus: probed [ 2.910616] usbcore: registered new interface driver usbserial_generic [ 2.913469] usbserial: USB Serial support registered for generic [ 2.916283] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.920638] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.922669] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.925637] mousedev: PS/2 mouse device common for all mice [ 2.929820] rtc_cmos 00:05: RTC can wake from S4 [ 2.932695] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.933037] rtc_cmos 00:05: registered as rtc0 [ 2.937937] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.941224] intel_pstate: CPU model not supported [ 2.944358] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.945950] hid: raw HID events driver (C) Jiri Kosina [ 2.950694] usbcore: registered new interface driver usbhid [ 2.950863] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.952804] usbhid: USB HID core driver [ 2.952987] drop_monitor: Initializing network drop monitor service [ 2.962510] Initializing XFRM netlink socket [ 2.964555] NET: Registered protocol family 10 [ 2.968784] Segment Routing with IPv6 [ 2.970356] NET: Registered protocol family 17 [ 2.972537] mpls_gso: MPLS GSO support [ 2.977834] RAS: Correctable Errors collector initialized. [ 2.980112] AVX version of gcm_enc/dec engaged. [ 2.981656] AES CTR mode by8 optimization enabled [ 3.064416] sched_clock: Marking stable (3064391874, 0)->(3994454897, -930063023) [ 3.067458] registered taskstats version 1 [ 3.069293] Loading compiled-in X.509 certificates [ 3.071227] zswap: loaded using pool lzo/zbud [ 3.096234] Key type big_key registered [ 3.110169] Key type encrypted registered [ 3.111656] ima: No TPM chip found, activating TPM-bypass! [ 3.113392] ima: Allocated hash algorithm: sha1 [ 3.114696] ima: No architecture policies found [ 3.116592] evm: Initialising EVM extended attributes: [ 3.118624] evm: security.selinux [ 3.119915] evm: security.ima [ 3.121188] evm: security.capability [ 3.122808] evm: HMAC attrs: 0x1 [ 3.125506] rtc_cmos 00:05: setting system clock to 2026-09-08 05:25:41 UTC (1788845141) [ 3.132127] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 3.134775] debug: unmapping init [mem 0xffffffff94982000-0xffffffff94c58fff] [ 3.141408] Write protecting the kernel read-only data: 28672k [ 3.144531] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.147106] debug: unmapping init [mem 0xffffffff93914000-0xffffffff939fffff] [ 3.178079] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.185095] systemd[1]: Detected virtualization kvm. [ 3.186876] systemd[1]: Detected architecture x86-64. [ 3.188455] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.216262] systemd[1]: No hostname configured. [ 3.217931] systemd[1]: Set hostname to . [ 3.219649] random: systemd: uninitialized urandom read (16 bytes read) [ 3.221948] systemd[1]: Initializing machine ID from random generator. [ 3.280783] random: ln: uninitialized urandom read (6 bytes read) [ 3.373562] random: systemd: uninitialized urandom read (16 bytes read) [ 3.377544] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.383461] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.387729] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.024195] device-mapper: uevent: version 1.0.3 [ 4.026570] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.825889] virtio_net virtio0 ens2: renamed from eth0 [ 4.879249] scsi host0: ata_piix [ 4.999960] scsi host1: ata_piix [ 5.001833] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.005162] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.633675] random: crng init done [ 9.643717] random: 7 urandom warning(s) missed due to ratelimiting [ 9.780806] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.116831] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.604250] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.328867] SELinux: Disabled at runtime. [ 15.447824] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.461720] systemd[1]: Detected virtualization kvm. [ 15.468445] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.316795] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.322556] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.341471] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.347836] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.359065] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.378263] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.395712] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 17.819326] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.836379] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.866622] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.895926] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.251896] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.325723] EDAC sbridge: Ver: 1.1.2 [ 24.632043] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.363239] NFS: Registering the id_resolver key type [ 25.366126] Key type id_resolver registered [ 25.370208] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg356-client login: [ 54.448764] hrtimer: interrupt took 18754466 ns [ 95.085656] libcfs: loading out-of-tree module taints kernel. [ 95.398057] Key type ._llcrypt registered [ 95.404522] Key type .llcrypt registered [ 95.936837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 95.963726] alg: No test for adler32 (adler32-zlib) [ 97.442679] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 98.628090] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 100.448038] Key type lgssc registered [ 102.658306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.965766] Lustre: Mounted lustre-client - version 2.17.57_108_g4a8ba26 [ 225.121472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.236436] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing check_logdir /tmp/testlogs/ [ 245.140290] Lustre: DEBUG MARKER: oleg356-client.virtnet: executing yml_node [ 245.740596] Lustre: lustre-OST0000-osc-ffff8c6e43cca000: disconnect after 24s idle [ 250.585204] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 253.409596] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 256.289307] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 257.827703] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 01:29:54 EDT 2026 [ 275.084611] Lustre: Unmounted lustre-client [ 302.079772] Key type lgssc unregistered [ 302.283947] LNet: 8412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.295261] LNetError: 8412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.328120] LNet: Removed LNI 192.168.203.56@tcp [ 303.026175] Key type .llcrypt unregistered [ 303.028337] Key type ._llcrypt unregistered [ 327.583235] Key type ._llcrypt registered [ 327.604093] Key type .llcrypt registered [ 327.942057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 327.973398] alg: No test for adler32 (adler32-zlib) [ 328.942523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 329.084814] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 329.106602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 333.289672] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 333.434424] LNet: 9283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.456150] LNetError: 9283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.486292] LNet: Removed LNI 192.168.203.56@tcp [ 334.447452] Key type .llcrypt unregistered [ 334.459486] Key type ._llcrypt unregistered [ 341.071395] Key type ._llcrypt registered [ 341.073141] Key type .llcrypt registered [ 342.572952] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:31:19 (1788845479) [ 343.093776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 343.119327] alg: No test for adler32 (adler32-zlib) [ 344.054571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.162639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.188635] LNet: 9701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.211466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.263569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 344.281910] LNet: 9708:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 351.484740] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:31:28 (1788845488) [ 351.753611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.785628] LNet: 10289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.809978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.880801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 358.527936] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:31:35 (1788845495) [ 358.683140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.729359] LNet: 10873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.782817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.876871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 366.389688] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:31:43 (1788845503) [ 366.539489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.562355] LNet: 11452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.609674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.738599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 375.023782] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:31:51 (1788845511) [ 375.127508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.172816] LNet: 12039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.209495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.370717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 376.004644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 384.433535] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:32:00 (1788845520) [ 384.559475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.594851] LNet: 12629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.649448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.786761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.265729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.895281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 392.111497] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:32:09 (1788845529) [ 392.296562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.370217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.428374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 400.053714] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:32:16 (1788845536) [ 400.349550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.417809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.471316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 400.509155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.059875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 401.105677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.719894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 401.783402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.460119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 402.515715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.077869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 403.134498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.768993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 403.802434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.260413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 404.307541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.908990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 404.953867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 405.610227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 405.681332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.216575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 406.257430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 406.838957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 406.891690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 407.300652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 407.325881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.672643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 407.740420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 415.046298] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:32:31 (1788845551) [ 415.173583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.210669] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.217466] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 415.252360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.394309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 415.446829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 422.324253] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:32:39 (1788845559) [ 422.518239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.566577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.679938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 422.720475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.267487] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:32:46 (1788845566) [ 429.482924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.548638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.668800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 429.710376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 436.718969] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:32:53 (1788845573) [ 437.027387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.087526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.180206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 437.239200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.110848] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:33:00 (1788845580) [ 444.212531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.258628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.347604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 444.390955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 451.953625] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:33:08 (1788845588) [ 452.039821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.069000] LNet: 17426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.074958] LNet: 17426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 452.114261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.210881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.288709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 459.715163] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:33:16 (1788845596) [ 459.826321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.866156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.975553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.597397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 467.725941] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:33:24 (1788845604) [ 467.855900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.901201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.132030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.402292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.085542] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:33:33 (1788845613) [ 476.258265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.354472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.512433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.715456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.142218] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:33:41 (1788845621) [ 485.334830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.401860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.201585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.679608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.086098] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:33:51 (1788845631) [ 495.214400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.271269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.403607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.506473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.054253] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:33:59 (1788845639) [ 503.373940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.443711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.556333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 503.628683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.178700] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:34:08 (1788845648) [ 511.574457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.634126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.714197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 520.201534] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:34:16 (1788845656) [ 520.391660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.465499] LNet: 22278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.476256] LNet: 22278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 520.524755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.667424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 528.425679] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:34:25 (1788845665) [ 528.598522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.697449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.823388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 529.612862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 536.393981] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:34:33 (1788845673) [ 536.523071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.584562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.711476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 536.752874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 544.557281] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:34:41 (1788845681) [ 544.756665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.818854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.885700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 544.940516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 552.053130] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:34:48 (1788845688) [ 552.135772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.177781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.275888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 552.350346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 560.079649] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:34:56 (1788845696) [ 560.177957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.235349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.360755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 560.407527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.722048] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:35:05 (1788845705) [ 568.854262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.920293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.986720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 569.036939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 569.051723] LNetError: 25819:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 569.086871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 569.112819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 576.230700] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:35:12 (1788845712) [ 576.366418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.431956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.570600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.insTiTADpf [ 576.721833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.insTiTADpf [ 576.817428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.insTiTADpf [ 583.742790] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:35:20 (1788845720) [ 583.793818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.844877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.903670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.958559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 584.024505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 590.123032] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:35:27 (1788845727) [ 590.536240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.607400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.715896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.744858] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 592.330600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 593.698611] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 595.623260] Key type lgssc registered [ 596.866931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 600.935778] LNetError: 27581:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.56@tcp99 has no available nets [ 603.749219] Key type lgssc unregistered [ 604.135639] LNetError: 28687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.158369] LNet: Removed LNI 192.168.203.56@tcp [ 605.801686] Key type .llcrypt unregistered [ 605.810720] Key type ._llcrypt unregistered [ 612.275492] Key type ._llcrypt registered [ 612.278882] Key type .llcrypt registered [ 613.540052] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:35:50 (1788845750) [ 614.074650] Key type .llcrypt unregistered [ 614.077094] Key type ._llcrypt unregistered [ 620.292136] Key type ._llcrypt registered [ 620.298717] Key type .llcrypt registered [ 620.499540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 620.521232] alg: No test for adler32 (adler32-zlib) [ 621.475020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.545412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 621.562789] LNet: 29596:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 621.568323] LNet: 29596:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 621.578717] LNet: 29596:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 621.604614] LNet: Message forwarding will be enabled in 2s [ 623.647586] LNet: Message forwarding enabled [ 624.020809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 624.047220] LNet: Message forwarding will stop in 2s to 6s [ 627.747667] LNet: Message forwarding disabled [ 628.576443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.613041] LNet: 29685:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 628.617347] LNet: 29685:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 628.628322] LNet: 29685:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 628.643963] LNet: Message forwarding will be enabled in 2s [ 628.796144] LNet: 29698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.492607] Key type .llcrypt unregistered [ 629.497260] Key type ._llcrypt unregistered [ 640.084718] Key type ._llcrypt registered [ 640.086711] Key type .llcrypt registered [ 641.374849] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:36:18 (1788845778) [ 641.822324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 641.838185] alg: No test for adler32 (adler32-zlib) [ 642.835402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.958313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 650.228246] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:36:26 (1788845786) [ 650.343791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.398890] LNet: 31092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.437814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.503700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 650.542544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 650.587950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 650.625210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 650.674795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 650.714626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 650.766677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 650.804204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 650.853389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 650.901614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 650.956588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 651.008456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 651.051885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 651.095346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 651.134818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 651.182894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 651.227628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 651.267553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.302920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 651.329858] LNetError: 31142:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 658.923907] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:36:35 (1788845795) [ 659.023255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.069347] LNet: 31708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.112383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.236892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.529084] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:36:43 (1788845803) [ 668.644650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg356-server [ 670.459295] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:36:47 (1788845807) [ 670.750726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.771681] LNet: 32646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.790924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.879031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.948345] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 672.141770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[57-61]@tcp [ 672.183425] LNet: 32779:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 672.255184] LNetError: 32650:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 672.286942] LNetError: 32650:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.58@tcp has gone from up to down [ 675.363811] LNetError: 32657:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 675.386275] LNetError: 32657:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 678.728431] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:36:55 (1788845815) [ 680.598191] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg356-server [ 682.403479] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:36:59 (1788845819) [ 682.557446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.584230] LNet: 33715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.595474] LNetError: 33715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.625772] LNet: Removed LNI 192.168.203.56@tcp [ 682.650640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.699372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.724177] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 683.352605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 683.378521] LNet: 33834:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 683.387880] LNet: 33834:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 683.412749] LNetError: 33721:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 683.419772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[57-62/2]@tcp [ 683.437544] LNetError: 33721:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 683.437600] LNetError: 33721:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.57@tcp has gone from up to down [ 689.248697] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:37:06 (1788845826) [ 689.337313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.372854] LNet: 34421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.381676] LNetError: 34421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.407479] LNet: Removed LNI 192.168.203.56@tcp [ 690.440875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.543542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 690.591356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.625894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.682455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.733162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.771576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.828331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 690.930181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.978268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.079945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.173256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.213916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.273908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 691.337875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.373472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.457223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.545339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.587394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.633498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 691.722970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.775850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.849030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 691.934848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.972305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.026417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 692.112891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.163551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.238708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 692.315176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.347608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.430905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 692.507687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.584211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.676872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 699.835514] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:37:16 (1788845836) [ 701.882781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg356-server [ 703.668881] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:37:20 (1788845840) [ 705.072720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg356-server [ 706.548958] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:37:23 (1788845843) [ 706.719448] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.724305] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 707.255797] Key type .llcrypt unregistered [ 707.257376] Key type ._llcrypt unregistered [ 712.471870] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.676809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.680510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.684305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.322990] Key type ._llcrypt registered [ 715.326134] Key type .llcrypt registered [ 715.562408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.581685] alg: No test for adler32 (adler32-zlib) [ 716.536925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.600074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.639364] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 717.250693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.267502] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.815917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 718.836802] LNet: Removed LNI 10.1.2.3@tcp [ 718.934810] LNet: 36541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.940460] LNetError: 36541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.950345] LNet: Removed LNI 192.168.203.56@tcp [ 719.403150] Key type .llcrypt unregistered [ 719.405048] Key type ._llcrypt unregistered [ 729.644886] Key type ._llcrypt registered [ 729.649200] Key type .llcrypt registered [ 731.262162] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:37:47 (1788845867) [ 731.913138] Key type .llcrypt unregistered [ 731.919349] Key type ._llcrypt unregistered [ 738.175211] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.326920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.329581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.333445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.171360] Key type ._llcrypt registered [ 741.178997] Key type .llcrypt registered [ 741.424184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.455339] alg: No test for adler32 (adler32-zlib) [ 742.456527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.528278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.551639] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 743.472660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.494825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.711322] LNet: 38029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.721421] LNetError: 38029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 746.728795] LNet: Removed LNI 10.1.2.3@tcp [ 746.748362] LNet: Removed LNI 192.168.203.56@tcp [ 747.558820] Key type .llcrypt unregistered [ 747.564512] Key type ._llcrypt unregistered [ 759.068096] Key type ._llcrypt registered [ 759.070464] Key type .llcrypt registered [ 760.199342] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:38:17 (1788845897) [ 760.743300] Key type .llcrypt unregistered [ 760.745692] Key type ._llcrypt unregistered [ 768.846622] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.924791] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.928041] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.939247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.697221] Key type ._llcrypt registered [ 771.698981] Key type .llcrypt registered [ 772.009126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.037519] alg: No test for adler32 (adler32-zlib) [ 772.986980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.050545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 773.111051] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 775.549300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.569270] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.227791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 778.278335] LNet: Removed LNI 10.1.2.3@tcp [ 778.306288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 778.338216] LNet: Removed LNI 10.1.2.31@tcp [ 778.356896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.390824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 779.814662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 779.847522] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 781.147742] LNet: 39945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.153978] LNetError: 39945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 781.160654] LNet: Removed LNI 10.1.2.31@tcp [ 781.181267] LNet: Removed LNI 10.1.2.3@tcp [ 781.783337] Key type .llcrypt unregistered [ 781.785706] Key type ._llcrypt unregistered [ 792.768610] Key type ._llcrypt registered [ 792.773063] Key type .llcrypt registered [ 794.323542] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:38:50 (1788845930) [ 794.690224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 794.712625] alg: No test for adler32 (adler32-zlib) [ 795.700471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.753995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.779908] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 796.436484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.457621] LNet: 40895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.463809] LNetError: 40895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.481971] LNet: Removed LNI 192.168.203.56@tcp [ 796.506315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.581488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.56@tcp [ 796.624967] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 797.306443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.56@tcp [ 797.357339] LNet: Removed LNI 192.168.203.56@tcp [ 797.445434] LNet: 41030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.994682] Key type .llcrypt unregistered [ 797.999167] Key type ._llcrypt unregistered [ 808.214815] Key type ._llcrypt registered [ 808.217738] Key type .llcrypt registered [ 809.416212] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:39:06 (1788845946) [ 811.202602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg356-server [ 813.420185] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:39:10 (1788845950) [ 813.885455] Key type .llcrypt unregistered [ 813.888239] Key type ._llcrypt unregistered [ 819.604100] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.751765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.755379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.759290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 822.256071] Key type ._llcrypt registered [ 822.260514] Key type .llcrypt registered [ 822.458059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.469450] alg: No test for adler32 (adler32-zlib) [ 823.386573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.468686] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 824.249676] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 826.491316] LNet: 42861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.497351] LNetError: 42861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.508432] LNet: Removed LNI 192.168.203.56@tcp1 [ 826.517354] LNet: Removed LNI 10.1.2.3@tcp2 [ 827.038437] Key type .llcrypt unregistered [ 827.042436] Key type ._llcrypt unregistered [ 836.838836] Key type ._llcrypt registered [ 836.842127] Key type .llcrypt registered [ 837.848916] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:39:35 (1788845975) [ 838.144470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.165400] alg: No test for adler32 (adler32-zlib) [ 839.082587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.150746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.172777] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 840.101325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.121326] LNet: 43827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.128612] LNetError: 43827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.152133] LNet: Removed LNI 192.168.203.56@tcp [ 840.177988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.249467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 840.274979] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 846.689184] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:39:43 (1788845983) [ 846.752941] LNet: 44410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.774622] LNetError: 44410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.788985] LNet: Removed LNI 192.168.203.56@tcp [ 847.546560] Key type .llcrypt unregistered [ 847.550793] Key type ._llcrypt unregistered [ 853.262428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 853.366525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 853.369697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 853.373432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.878582] Key type ._llcrypt registered [ 855.882807] Key type .llcrypt registered [ 856.030856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.042397] alg: No test for adler32 (adler32-zlib) [ 856.993547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.059969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.092091] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 857.931775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 857.960481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.417569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.439807] LNet: 45158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.448466] LNetError: 45158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 860.457294] LNet: Removed LNI 10.1.2.3@tcp [ 860.470041] LNet: Removed LNI 192.168.203.56@tcp [ 860.504743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.592303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 860.623126] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 860.626931] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.180153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.207771] LNet: 45185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.219327] LNetError: 45185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.232674] LNetError: 45185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 861.248794] LNet: Removed LNI 10.1.2.3@tcp [ 861.264417] LNet: Removed LNI 192.168.203.56@tcp [ 861.299485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.377742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 861.402206] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 861.411673] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.880978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 861.928122] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 862.892905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.916597] LNet: 45347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.929285] LNetError: 45347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 862.934961] LNetError: 45347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.940528] LNet: Removed LNI 10.1.2.3@tcp [ 862.943090] LNet: Removed LNI 192.168.203.56@tcp [ 862.966215] LNet: Removed LNI 192.168.203.56@tcp2 [ 862.993960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.094539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 863.131485] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 863.136534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.143256] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 870.191743] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:40:07 (1788846007) [ 870.296373] LNet: 45936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.319771] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 870.327239] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 870.338247] LNet: Removed LNI 10.1.2.3@tcp [ 870.346956] LNet: Removed LNI 192.168.203.56@tcp [ 870.382271] LNet: Removed LNI 192.168.203.56@tcp2 [ 870.924153] Key type .llcrypt unregistered [ 870.928117] Key type ._llcrypt unregistered [ 877.486810] Key type ._llcrypt registered [ 877.491943] Key type .llcrypt registered [ 877.711909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.742497] alg: No test for adler32 (adler32-zlib) [ 878.688744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.739346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.762625] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 879.561781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.576381] LNet: 46336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.585466] LNetError: 46336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.601404] LNet: Removed LNI 192.168.203.56@tcp [ 879.629880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.700800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 879.722626] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 880.288299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.325567] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.339711] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.358082] LNet: Removed LNI 192.168.203.56@tcp [ 880.375788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.446566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 880.479817] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 880.944679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.982197] LNet: Removed LNI 192.168.203.56@tcp [ 881.000697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.053548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 881.081392] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 881.517403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.632094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.637703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.647868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 884.177144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.203175] LNet: 46474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.209164] LNet: 46474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 884.214547] LNetError: 46474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.221507] LNetError: 46474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.238200] LNet: Removed LNI 192.168.203.56@tcp [ 884.271228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.385107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 884.415770] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 891.650763] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:40:28 (1788846028) [ 891.883573] LNet: 47072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.887160] LNetError: 47072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 891.896388] LNet: Removed LNI 192.168.203.56@tcp [ 892.641299] Key type .llcrypt unregistered [ 892.647938] Key type ._llcrypt unregistered [ 899.284830] Key type ._llcrypt registered [ 899.291891] Key type .llcrypt registered [ 899.514520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.532216] alg: No test for adler32 (adler32-zlib) [ 900.494530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.562493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.600684] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 901.716896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.762141] LNet: 47467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.774591] LNetError: 47467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.795268] LNet: Removed LNI 192.168.203.56@tcp [ 901.840325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.916154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 901.948349] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 908.473138] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:40:45 (1788846045) [ 908.859331] LNet: 48062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.871281] LNetError: 48062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.894970] LNet: Removed LNI 192.168.203.56@tcp [ 909.746739] Key type .llcrypt unregistered [ 909.752439] Key type ._llcrypt unregistered [ 917.040803] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.168269] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.176494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.189996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.834816] Key type ._llcrypt registered [ 919.841797] Key type .llcrypt registered [ 920.061531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.072442] alg: No test for adler32 (adler32-zlib) [ 920.933728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.025852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.051702] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 921.896467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.908297] LNet: 48510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.915670] LNetError: 48510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.933377] LNet: Removed LNI 192.168.203.56@tcp [ 921.963276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.053775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 922.091031] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 929.064391] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:41:05 (1788846065) [ 929.210340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.255253] LNet: 49101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.263440] LNetError: 49101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.278648] LNet: Removed LNI 192.168.203.56@tcp [ 929.304679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.437952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 937.320668] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:41:13 (1788846073) [ 937.461187] LNet: 49682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.341207] Key type .llcrypt unregistered [ 938.350338] Key type ._llcrypt unregistered [ 944.903700] Key type ._llcrypt registered [ 944.908690] Key type .llcrypt registered [ 945.327025] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.360670] alg: No test for adler32 (adler32-zlib) [ 946.407923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.473648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.513404] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 947.593972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.614650] LNet: 50084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.624950] LNetError: 50084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.645603] LNet: Removed LNI 192.168.203.56@tcp [ 947.675614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.810832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 947.857359] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 956.623419] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:41:33 (1788846093) [ 956.901916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.928813] LNet: 50678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.945177] LNetError: 50678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.964869] LNet: Removed LNI 192.168.203.56@tcp [ 956.990972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.056804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 963.801731] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:41:40 (1788846100) [ 965.762287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 967.945499] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:41:44 (1788846104) [ 969.334422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 971.699793] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:41:48 (1788846108) [ 973.755601] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 975.647498] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:41:52 (1788846112) [ 977.247201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 978.963691] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:41:56 (1788846116) [ 980.398386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 982.173775] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:41:59 (1788846119) [ 982.529899] LNet: 52734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.141443] Key type .llcrypt unregistered [ 983.146206] Key type ._llcrypt unregistered [ 989.639332] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 989.909967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 989.929972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 989.941092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 992.717459] Key type ._llcrypt registered [ 992.725369] Key type .llcrypt registered [ 992.913788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.926299] alg: No test for adler32 (adler32-zlib) [ 993.858739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.944040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 993.973917] LNet: Added LNI 192.168.203.56@tcp [8/512/0/180] [ 993.982834] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 993.988601] LNet: 53049:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 994.100804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.121526] LNet: 53068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.131346] LNetError: 53068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.145930] LNet: Removed LNI 10.1.2.3@tcp [ 994.152801] LNetError: 53068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.169921] LNet: Removed LNI 192.168.203.56@tcp [ 994.188406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.250474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 994.267966] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 994.365700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.415623] LNet: 53092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.422587] LNetError: 53092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.442609] LNet: Removed LNI 192.168.203.56@tcp [ 994.458959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.571730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 994.592789] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 994.603683] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.611166] LNet: 53100:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 994.755581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.777628] LNet: 53118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.791762] LNetError: 53118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.805786] LNet: Removed LNI 10.1.2.3@tcp [ 994.813948] LNetError: 53118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.841851] LNet: Removed LNI 192.168.203.56@tcp [ 994.881414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.974578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 995.015629] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 995.023625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 995.036215] LNet: 53126:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 995.243747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.263727] LNet: 53152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.268882] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 995.274910] LNet: Removed LNI 10.1.2.3@tcp [ 995.278992] LNetError: 53152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.293851] LNet: Removed LNI 192.168.203.56@tcp [ 995.315216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.395961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 995.457975] LNet: Added LNI 192.168.203.56@tcp [8/256/0/182] [ 995.467284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 995.601798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 995.624963] LNet: 53180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.638645] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 995.644643] LNet: Removed LNI 10.1.2.3@tcp [ 995.651210] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 995.675386] LNet: Removed LNI 192.168.203.56@tcp [ 995.717816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.801163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 995.837562] LNet: Added LNI 192.168.203.56@tcp [8/256/0/182] [ 995.842141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 995.848402] LNet: 53188:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 995.919807] LNet: Removed LNI 10.1.2.3@tcp [ 995.930530] LNet: Removed LNI 192.168.203.56@tcp [ 996.729283] Key type .llcrypt unregistered [ 996.732799] Key type ._llcrypt unregistered [ 1008.350115] Key type ._llcrypt registered [ 1008.352076] Key type .llcrypt registered [ 1009.776988] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:42:26 (1788846146) [ 1011.472348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1013.546244] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:42:30 (1788846150) [ 1013.920593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.938860] alg: No test for adler32 (adler32-zlib) [ 1014.887186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.947833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1014.974585] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1015.801475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.56 [ 1015.832608] LNet: 54449:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1015.842767] LNet: Removed LNI 192.168.203.56@tcp [ 1021.934074] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:42:38 (1788846158) [ 1022.092414] LNet: 55012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.104717] LNetError: 55012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.123609] LNet: Removed LNI 192.168.203.56@tcp [ 1022.981250] Key type .llcrypt unregistered [ 1022.983695] Key type ._llcrypt unregistered [ 1029.577302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1029.797582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1032.650861] Key type ._llcrypt registered [ 1032.652704] Key type .llcrypt registered [ 1032.788089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1032.802673] alg: No test for adler32 (adler32-zlib) [ 1034.226389] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1040.112524] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:42:57 (1788846177) [ 1040.225101] LNet: 55932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.236505] LNetError: 55932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.267383] LNet: Removed LNI 10.1.2.3@tcp [ 1040.948567] Key type .llcrypt unregistered [ 1040.954705] Key type ._llcrypt unregistered [ 1046.655299] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1046.827674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1046.831349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1046.838725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.567704] Key type ._llcrypt registered [ 1049.570680] Key type .llcrypt registered [ 1049.746280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.758487] alg: No test for adler32 (adler32-zlib) [ 1050.781056] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1056.875950] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:43:13 (1788846193) [ 1058.174915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1060.229812] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:43:17 (1788846197) [ 1060.317632] LNet: 57114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.326880] LNetError: 57114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1060.343433] LNet: Removed LNI 10.1.2.3@tcp [ 1061.216450] Key type .llcrypt unregistered [ 1061.220201] Key type ._llcrypt unregistered [ 1066.399758] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1066.636840] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1066.642056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1066.647276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1069.372419] Key type ._llcrypt registered [ 1069.374788] Key type .llcrypt registered [ 1069.513714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.525870] alg: No test for adler32 (adler32-zlib) [ 1070.435934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.491536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1079.010410] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:43:35 (1788846215) [ 1079.599370] LNet: 58349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.605359] LNetError: 58349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1079.626904] LNet: Removed LNI 10.1.2.3@tcp [ 1080.552457] Key type .llcrypt unregistered [ 1080.554831] Key type ._llcrypt unregistered [ 1087.157750] Key type ._llcrypt registered [ 1087.160453] Key type .llcrypt registered [ 1087.427769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.471370] alg: No test for adler32 (adler32-zlib) [ 1088.405679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1088.442651] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1095.976301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1096.838959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1096.879904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1096.920343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1096.943956] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1097.175590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.232623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.274798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.304208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.333753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.376293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.413216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.464797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.490566] LNet: 58800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.522873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.562237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.607373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.641634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.690279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.697831] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1097.698540] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1097.732349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.770495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.244393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1098.286317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1098.335517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1098.635698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1098.678701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1098.715908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.759971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.818369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.857731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.890616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.920490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.956164] LNet: 58909:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.990318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.023734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.065658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.101595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.132774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.172257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.201480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.970607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1100.048126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1100.113734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1100.157795] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1100.171557] LNet: Skipped 1 previous similar message [ 1100.494807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.527200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.575595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.625358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.677983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.717870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.776739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.828993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.858392] LNet: 59018:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.888887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.922623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.982582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.034797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.113927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.172562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.226780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1102.048168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1102.123656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1102.203313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1102.252317] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1102.624558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.665209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.706510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.746934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.773949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.808884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.837677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.902354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.935540] LNet: 59128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.958205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.013316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.055952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.108412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.154774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.194606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.228772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.678390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1103.709263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1103.739779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1104.017863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.048688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.070164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.097449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.135498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.205417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.262428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.300226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.338583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.403226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.438860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.485429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.537038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.575620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.604963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1105.203375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1105.238307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1105.267524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1105.469504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.497260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.520492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.545164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.577527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.608243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.635064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.657868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.690458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.719920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.770944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.816808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.859854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.897155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.938818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.273762] LNet: 59387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.280864] LNetError: 59387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.299276] LNet: Removed LNI 192.168.203.56@tcp [ 1106.720591] Key type .llcrypt unregistered [ 1106.728136] Key type ._llcrypt unregistered [ 1125.602331] Key type ._llcrypt registered [ 1125.607466] Key type .llcrypt registered [ 1127.293913] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:44:23 (1788846263) [ 1128.342540] Key type .llcrypt unregistered [ 1128.343838] Key type ._llcrypt unregistered [ 1135.768524] Key type ._llcrypt registered [ 1135.774519] Key type .llcrypt registered [ 1136.242501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1136.310365] alg: No test for adler32 (adler32-zlib) [ 1137.318326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1137.358631] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1146.258231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1158.380814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1158.424532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1158.453883] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1160.506602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1160.563846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1160.643702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1160.736728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1160.814140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 1 [ 1160.875128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 1 [ 1160.929963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 1 [ 1160.967990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 1 [ 1161.005371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1161.023871] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1161.029664] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1161.306853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.355759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.411442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.474859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.526299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.572203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.614970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.671610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.705529] LNet: 60853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.736358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.781885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.877167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.933987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.996685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.041915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.083855] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1162.095331] LNet: Skipped 3 previous similar messages [ 1162.122565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.824924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1162.862657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1162.917180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1162.962860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1162.997617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 1 [ 1163.038365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 1 [ 1163.101878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 1 [ 1163.165898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 1 [ 1163.242965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1163.273602] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1163.557904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.592469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.644591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.698541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.735053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.777173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.823455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.854870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.881282] LNet: 60975:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.929321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.969281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.004413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.038685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.077605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.145618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.205112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.769755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1164.831217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1164.877721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1164.938886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1164.981163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 1 [ 1165.028584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 1 [ 1165.059366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 1 [ 1165.102364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 1 [ 1165.135367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1165.418528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.441726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.466205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.499237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.523602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.555403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.580878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.608742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.629928] LNet: 61096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.645985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.686221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.717818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.771714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.816055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.866507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.910305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.485283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1166.530182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1166.603596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1166.633235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1166.674749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 1 [ 1166.714868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 1 [ 1166.751569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 1 [ 1166.802573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 1 [ 1166.858681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1166.890118] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1166.895151] LNet: Skipped 5 previous similar messages [ 1167.122250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.179496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.224343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.275716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.333865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.388220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.438360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.524110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.596148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.652766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.717165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.780239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.823296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.875486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.915734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.440798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1168.477890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1168.507794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1168.537682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1168.569427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 1 [ 1168.602979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 1 [ 1168.635309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 1 [ 1168.676619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 1 [ 1168.731649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1169.004838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.035419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.064886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.090691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.118258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.147826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.185423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.223542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.237426] LNet: 61339:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.245049] LNet: 61339:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1169.277821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.319755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.361628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.411220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.487623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.549497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.623913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.230357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1170.271886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1170.324688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1170.354790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1170.385722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_error -r 1 [ 1170.431185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_error -r 1 [ 1170.465531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_error -r 1 [ 1170.505321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_error -r 1 [ 1170.552942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1170.839344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.867907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.910045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.952128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.014068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.067153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.119768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.165863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.213862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.269794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.337067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.374411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.408659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.441970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.483340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.550430] LNet: 61536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.560770] LNetError: 61536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.566542] LNet: Removed LNI 192.168.203.56@tcp [ 1178.598836] LNet: Removed LNI 192.168.203.56@tcp1 [ 1179.262461] Key type .llcrypt unregistered [ 1179.266161] Key type ._llcrypt unregistered [ 1198.751428] Key type ._llcrypt registered [ 1198.753913] Key type .llcrypt registered [ 1200.087257] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:45:36 (1788846336) [ 1201.182173] Key type .llcrypt unregistered [ 1201.189110] Key type ._llcrypt unregistered [ 1209.345829] Key type ._llcrypt registered [ 1209.348111] Key type .llcrypt registered [ 1209.632942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1209.659320] alg: No test for adler32 (adler32-zlib) [ 1210.555307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1210.585292] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1217.832957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1218.682697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1218.748884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1218.806396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1218.830957] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1219.180910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.213626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.250504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.318263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.365670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.430757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.496478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.549799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.605691] LNet: 62781:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1219.644205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.706678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.768434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.842315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.894358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.933543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.986988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1220.639076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1220.695620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1220.741974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1220.769544] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1221.087458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1221.147489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1221.210643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1221.252229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.295253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.336989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.368339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.401142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.427722] LNet: 62891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.454667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.510955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.556035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.595478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.629703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.674226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.720744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1222.313761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1222.356617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1222.423988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1222.460415] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1222.709659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.756493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.792975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.833370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.862856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.891502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.932620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.977993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1223.029989] LNet: 63000:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1223.084452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1223.128825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1223.185963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1223.233942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1223.327292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1223.388461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1223.453909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1223.942228] LNet: 63041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1223.955076] LNetError: 63041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1223.987301] LNet: Removed LNI 192.168.203.56@tcp [ 1224.440539] Key type .llcrypt unregistered [ 1224.443742] Key type ._llcrypt unregistered [ 1243.154850] Key type ._llcrypt registered [ 1243.156626] Key type .llcrypt registered [ 1245.022099] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:46:21 (1788846381) [ 1246.512425] Key type .llcrypt unregistered [ 1246.514239] Key type ._llcrypt unregistered [ 1254.511869] Key type ._llcrypt registered [ 1254.519490] Key type .llcrypt registered [ 1254.934382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1254.958161] alg: No test for adler32 (adler32-zlib) [ 1256.048109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1256.168180] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1266.446478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1280.051423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.124756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.157016] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1282.427771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1282.478254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1282.514586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1282.552553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1282.635691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 1 [ 1282.685744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 1 [ 1282.731992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 1 [ 1282.790859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 1 [ 1282.843759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1282.880819] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1282.890889] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1283.275919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.325460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.367731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.435865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.478390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.541101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.582723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.625916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.654071] LNet: 64504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.677415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.718317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.765570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.812427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.859429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.899563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.934721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1283.973822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.527345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1284.570075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1284.626993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1284.664330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1284.692585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_error -r 1 [ 1284.725261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_error -r 1 [ 1284.764495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_error -r 1 [ 1284.805359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_error -r 1 [ 1284.865883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1284.913742] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1284.919428] LNet: Skipped 2 previous similar messages [ 1285.206273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.236473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1285.275678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.324250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.355930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.395935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.450797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.504852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.525556] LNet: 64627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.555355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.598280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.634791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.660181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.682578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.714967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.737518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1285.764552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.263558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1286.308905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1286.348924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1286.412490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1286.454494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_timeout -r 1 [ 1286.485514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_timeout -r 1 [ 1286.520928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_timeout -r 1 [ 1286.563370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_timeout -r 1 [ 1286.620286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.156@tcp [ 1286.648567] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1286.887519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.921067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.953614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.978462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.004776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.037689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.066442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1287.102876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1287.124401] LNet: 64751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1287.145313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1287.186802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1287.235378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1287.291278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.321315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.376267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1287.419960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1287.459821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.779301] LNet: 64828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.784649] LNetError: 64828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.797545] LNet: Removed LNI 192.168.203.56@tcp [ 1292.811777] LNet: Removed LNI 192.168.203.56@tcp1 [ 1293.349637] Key type .llcrypt unregistered [ 1293.351403] Key type ._llcrypt unregistered [ 1311.323772] Key type ._llcrypt registered [ 1311.330410] Key type .llcrypt registered [ 1313.107386] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:47:29 (1788846449) [ 1314.754270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1316.849447] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:47:33 (1788846453) [ 1317.863403] Key type .llcrypt unregistered [ 1317.865237] Key type ._llcrypt unregistered [ 1324.084711] Key type ._llcrypt registered [ 1324.088955] Key type .llcrypt registered [ 1324.356202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.385355] alg: No test for adler32 (adler32-zlib) [ 1325.392926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1325.430671] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1332.819947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1333.684191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1333.769353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1333.848833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1333.880172] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1334.095580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.143749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.200200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.254974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.341078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.402147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.478845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.555118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.600738] LNet: 66367:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.628275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.684377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.687509] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1334.701375] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1334.760838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.838252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.900579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.960990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.367589] LNet: 66406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.377677] LNetError: 66406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.404212] LNet: Removed LNI 192.168.203.56@tcp [ 1336.004280] Key type .llcrypt unregistered [ 1336.007396] Key type ._llcrypt unregistered [ 1355.425451] Key type ._llcrypt registered [ 1355.428526] Key type .llcrypt registered [ 1355.741247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1355.770398] alg: No test for adler32 (adler32-zlib) [ 1356.791966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1356.848221] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1365.053780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1373.991525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.061340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.100331] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1375.665541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1375.720298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1375.780856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1375.856917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1375.933626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e network_timeout -r 1 [ 1375.990485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e network_timeout -r 1 [ 1376.040352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e network_timeout -r 1 [ 1376.067336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e network_timeout -r 1 [ 1376.092224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1376.108582] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1376.328311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.354471] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1376.359427] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1376.379441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.406577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.438850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.490872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.529763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.561769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.584378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.607976] LNet: 67280:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.632378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.676770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.719888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.766567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.820272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.872611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1381.597526] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1381.966366] LNet: 67354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.978864] LNetError: 67354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.990157] LNet: Removed LNI 192.168.203.56@tcp [ 1382.012804] LNet: Removed LNI 192.168.203.56@tcp1 [ 1382.520781] Key type .llcrypt unregistered [ 1382.524183] Key type ._llcrypt unregistered [ 1398.616758] Key type ._llcrypt registered [ 1398.622112] Key type .llcrypt registered [ 1400.338616] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:48:56 (1788846536) [ 1400.808675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.825917] alg: No test for adler32 (adler32-zlib) [ 1401.798656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.885330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.910727] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1402.757447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1402.788780] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1403.603204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1403.692754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1403.789603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1403.835872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1403.937111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1403.987868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1404.048774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1404.074719] LNetError: 68451:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1404.895333] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1404.898373] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1407.007838] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1407.574905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1411.104749] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1412.232688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1412.267376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1412.313639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.332956] LNet: 68568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.339439] LNetError: 68568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.346026] LNet: Removed LNI 192.168.203.56@tcp [ 1412.366379] LNet: Removed LNI 192.168.203.56@tcp1 [ 1412.384962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.434495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.453786] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1413.067190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.087496] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1413.954360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1413.985393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.075459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1414.136898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1414.178373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1414.213390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1414.274676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.302829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1414.336158] LNetError: 68841:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1414.504569] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1416.762541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1420.640386] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1420.647333] LNet: Skipped 1 previous similar message [ 1425.981646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1426.005615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1426.029222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.654462] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:49:30 (1788846570) [ 1433.835847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.894919] LNet: 69552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.909125] LNetError: 69552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.917114] LNetError: 69552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.928816] LNet: Removed LNI 192.168.203.56@tcp [ 1433.944888] LNet: Removed LNI 192.168.203.56@tcp1 [ 1433.992354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.082330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.107959] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1435.032354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1435.056698] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1436.009099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1436.085527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1436.111547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1436.145667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1436.185103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1436.223408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1436.253676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1436.395452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.087334] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.56@tcp [ 1437.088721] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1437.118930] LNet: Skipped 1 previous similar message [ 1451.806911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1452.015719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1452.058597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.100794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 500 [ 1452.211214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1459.007476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1459.127696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.165397] LNet: 70050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.196678] LNetError: 70050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.200307] LNetError: 70050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1459.209444] LNet: Removed LNI 192.168.203.56@tcp [ 1459.218970] LNet: Removed LNI 192.168.203.56@tcp1 [ 1459.262587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.346239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.388436] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1460.392919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1460.427339] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1461.650899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1461.723428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.814211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1461.846811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1461.871363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1461.900778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1461.932536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1461.968227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.56@tcp --health 0 [ 1462.375481] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1462.388197] LNet: Skipped 3 previous similar messages [ 1463.396868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1472.775749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1472.819537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1472.901661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.688227] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:50:17 (1788846617) [ 1481.205537] LNet: 71085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.240420] LNetError: 71085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.246654] LNetError: 71085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.261219] LNet: Removed LNI 192.168.203.56@tcp [ 1481.272366] LNet: Removed LNI 192.168.203.56@tcp1 [ 1482.263274] Key type .llcrypt unregistered [ 1482.265141] Key type ._llcrypt unregistered [ 1489.235516] Key type ._llcrypt registered [ 1489.237933] Key type .llcrypt registered [ 1489.522839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1489.550769] alg: No test for adler32 (adler32-zlib) [ 1490.508698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1490.560891] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1499.935236] Lustre: DEBUG MARKER: Initial discovery [ 1499.997192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1502.685844] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1502.817960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1502.899926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1502.940766] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1502.991129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1503.034039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -l 3 [ 1503.080234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.156@tcp -r 1 -m GET -e local_error [ 1503.119773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.156@tcp -r 1 -m PUT -l 6 [ 1503.906574] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1505.008555] Lustre: DEBUG MARKER: Force oleg356-client.virtnet to discover 192.168.203.156@tcp (in background) [ 1505.094412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.156@tcp [ 1507.101304] Lustre: DEBUG MARKER: Force oleg356-server to discover 192.168.203.56@tcp [ 1508.199716] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1509.863826] Lustre: DEBUG MARKER: Wait for 71837 [ 1511.605396] Lustre: DEBUG MARKER: Finished wait on 71837 [ 1511.738136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1511.837447] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1511.851565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1512.198343] LNet: 72295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.206705] LNetError: 72295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.238061] LNet: Removed LNI 192.168.203.56@tcp [ 1512.899871] Key type .llcrypt unregistered [ 1512.901718] Key type ._llcrypt unregistered [ 1529.691691] Key type ._llcrypt registered [ 1529.693768] Key type .llcrypt registered [ 1531.302739] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:51:07 (1788846667) [ 1532.033528] Key type .llcrypt unregistered [ 1532.035882] Key type ._llcrypt unregistered [ 1538.893256] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1539.183587] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1539.194510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1539.200128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1541.766535] Key type ._llcrypt registered [ 1541.768133] Key type .llcrypt registered [ 1541.972397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1541.985821] alg: No test for adler32 (adler32-zlib) [ 1542.936520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.977589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.002743] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1544.015680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1544.045042] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1554.631838] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:51:31 (1788846691) [ 1554.744479] LNet: 74356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.752405] LNetError: 74356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1554.778536] LNet: Removed LNI 10.1.2.3@tcp [ 1554.823768] LNet: Removed LNI 192.168.203.56@tcp [ 1555.506546] Key type .llcrypt unregistered [ 1555.508772] Key type ._llcrypt unregistered [ 1562.721428] Key type ._llcrypt registered [ 1562.723096] Key type .llcrypt registered [ 1563.028522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.068369] alg: No test for adler32 (adler32-zlib) [ 1564.000154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.073203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1564.092469] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1564.906226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1564.934909] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1566.261610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.56@tcp [ 1567.327595] LNetError: Unexpected error -22 connecting to 192.168.203.56@tcp at host 192.168.203.56:988 [ 1573.984580] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:51:50 (1788846710) [ 1574.107944] LNet: 75486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.121796] LNetError: 75486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1574.131852] LNet: Removed LNI 10.1.2.3@tcp [ 1574.152515] LNet: Removed LNI 192.168.203.56@tcp [ 1575.066712] Key type .llcrypt unregistered [ 1575.071887] Key type ._llcrypt unregistered [ 1582.484090] Key type ._llcrypt registered [ 1582.486108] Key type .llcrypt registered [ 1582.682751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1582.699638] alg: No test for adler32 (adler32-zlib) [ 1583.690481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.746562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1583.779388] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1584.505175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1584.526520] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 1585.430444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.56@tcp1 --nid 192.168.203.56@tcp2 [ 1589.876904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp2 [ 1589.919913] LNetError: 76325:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.56@tcp1 to 192.168.203.56@tcp2 - Route table may be misconfigured [ 1589.965668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp2 192.168.203.56@tcp1 [ 1597.918646] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:52:14 (1788846734) [ 1598.055735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.086677] LNet: 76893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.099327] LNetError: 76893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.137751] LNet: Removed LNI 192.168.203.56@tcp1 [ 1599.144734] LNetError: 76893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.187647] LNet: Removed LNI 192.168.203.56@tcp2 [ 1599.220754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.278335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.305666] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1600.014790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1600.033746] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1600.946392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 1600.988755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.56@tcp [ 1601.023568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp1 192.168.203.56@tcp1 [ 1601.055873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp -e network_timeout [ 1601.084388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -e network_timeout [ 1601.132648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -e network_timeout [ 1601.175066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -e network_timeout [ 1601.208321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1601.227821] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1601.256091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1601.311101] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1601.399886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1601.559904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1607.559190] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:52:24 (1788846744) [ 1607.797414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.819176] LNet: 77760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.829755] LNetError: 77760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.834370] LNet: Removed LNI 192.168.203.56@tcp [ 1608.874736] LNet: Removed LNI 192.168.203.56@tcp1 [ 1608.912839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.041359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1609.300661] LNet: 77797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.683739] Key type .llcrypt unregistered [ 1609.687930] Key type ._llcrypt unregistered [ 1622.462664] Key type ._llcrypt registered [ 1622.464538] Key type .llcrypt registered [ 1624.316100] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:52:40 (1788846760) [ 1624.793417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1624.817616] alg: No test for adler32 (adler32-zlib) [ 1625.759268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.609819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1629.771829] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:52:46 (1788846766) [ 1629.915745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.940534] LNet: 78887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.987879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.082563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.122859] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1631.048669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1631.078923] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1632.165844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1632.199649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp1 [ 1632.274919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp1 [ 1638.912756] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:52:55 (1788846775) [ 1641.024784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg356-server [ 1643.034736] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:52:59 (1788846779) [ 1644.936208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg356-server [ 1646.884520] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:53:03 (1788846783) [ 1648.866115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg356-server [ 1650.934540] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:53:07 (1788846787) [ 1653.302836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg356-server [ 1655.690554] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:53:12 (1788846792) [ 1657.998031] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg356-server [ 1660.264625] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:53:16 (1788846796) [ 1662.688840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg356-server [ 1665.124074] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:53:21 (1788846801) [ 1667.398416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg356-server [ 1669.261822] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:53:26 (1788846806) [ 1670.860146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg356-server [ 1672.866774] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:53:29 (1788846809) [ 1674.854425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg356-server [ 1677.185853] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:53:33 (1788846813) [ 1677.274310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.302979] LNet: 82766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.308356] LNetError: 82766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.315232] LNet: Removed LNI 192.168.203.56@tcp [ 1678.374650] LNet: Removed LNI 192.168.203.56@tcp1 [ 1678.423817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.507697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.555845] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1679.522576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1679.639419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1679.869710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.901649] LNet: 82919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.913657] LNetError: 82919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.923371] LNetError: 82919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1679.941507] LNet: Removed LNI 192.168.203.56@tcp [ 1679.973267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.052905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1680.074179] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1681.051252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1681.354537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.390921] LNetError: 83064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1681.419424] LNet: Removed LNI 192.168.203.56@tcp [ 1681.457709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.549871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.586996] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1682.605593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1682.719323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1682.992960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.037555] LNet: 83212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.046743] LNet: 83212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1684.075951] LNet: Removed LNI 192.168.203.56@tcp [ 1684.120495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.224434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1684.269678] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1685.355714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1685.661315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.721848] LNetError: 83357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.734715] LNetError: 83357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1685.760362] LNet: Removed LNI 192.168.203.56@tcp [ 1685.784811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.865324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.911118] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1686.816761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1686.923220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1687.223246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.263078] LNet: 83504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.288045] LNet: 83504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1688.358507] LNet: Removed LNI 192.168.203.56@tcp [ 1688.385867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.466560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1688.520968] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1689.830212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1690.119560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.153220] LNetError: 83650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.158677] LNetError: 83650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1691.176800] LNet: Removed LNI 192.168.203.56@tcp [ 1691.191579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.288562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.333159] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1692.311029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1692.397233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1693.805678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.888165] LNet: Removed LNI 192.168.203.56@tcp [ 1694.914833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.062484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1695.109686] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1696.252402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1696.499526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.518262] LNet: 83952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.521594] LNet: 83952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1697.584735] LNet: Removed LNI 192.168.203.56@tcp [ 1697.619340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.691671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.729354] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1698.653762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1698.810536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1699.216691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.263577] LNetError: 84099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.274467] LNetError: 84099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1699.287928] LNet: Removed LNI 192.168.203.56@tcp [ 1699.365618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.484490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1699.530093] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1700.711763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1702.180665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.276849] LNet: Removed LNI 192.168.203.56@tcp [ 1703.308456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.431575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.469796] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1704.438290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1704.540636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1704.902763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.966331] LNet: Removed LNI 192.168.203.56@tcp [ 1706.027702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.085968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1706.105790] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1707.058689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1707.546813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.652578] LNet: Removed LNI 192.168.203.56@tcp [ 1708.694629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.791726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.827420] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1709.736425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1709.792250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.861128] LNet: Removed LNI 192.168.203.56@tcp [ 1709.929724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.025554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.073204] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1710.956720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1711.044588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.069039] LNet: Removed LNI 192.168.203.56@tcp [ 1711.087137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.194841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1711.298825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.356305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.420441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1711.541990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.597672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.683126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1711.763550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.803284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.857680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1711.880979] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1712.888261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.923510] LNet: 84987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.933445] LNet: 84987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1712.954725] LNet: Removed LNI 192.168.203.56@tcp [ 1712.968085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.075876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1721.233159] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:54:17 (1788846857) [ 1721.399791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.487686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.576621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1721.605551] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1723.193921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.241943] LNetError: 85707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.251874] LNetError: 85707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1723.283381] LNet: Removed LNI 192.168.203.56@tcp231 [ 1723.311241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.383534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1723.404887] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1723.411670] LNet: 85715:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.900364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1723.942876] LNet: Removed LNI 192.168.203.56@tcp231 [ 1723.962437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1723.986461] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1725.413882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.463355] LNet: Removed LNI 192.168.203.56@tcp231 [ 1725.495504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.580917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1725.639346] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1725.649811] LNet: 85867:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1726.296200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1726.335892] LNet: Removed LNI 192.168.203.56@tcp231 [ 1726.373924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1726.429514] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/0] [ 1727.773370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.834235] LNet: Removed LNI 192.168.203.56@tcp231 [ 1727.868355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.950909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1728.002435] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1728.020399] LNet: 86019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1728.698262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1728.731412] LNet: Removed LNI 192.168.203.56@tcp231 [ 1728.753422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1728.789475] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/60] [ 1729.976620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.030051] LNet: Removed LNI 192.168.203.56@tcp231 [ 1730.067918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.144743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1730.166610] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1730.177850] LNet: 86170:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1730.500897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1730.535642] LNet: Removed LNI 192.168.203.56@tcp231 [ 1730.552328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1730.576869] LNet: Added LNI 192.168.203.56@tcp231 [8/256/0/180] [ 1738.530705] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:54:35 (1788846875) [ 1738.833940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.867466] LNet: Removed LNI 192.168.203.56@tcp231 [ 1738.888912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.962529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1738.985216] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1739.671592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1739.708893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1739.732777] LNet: Removed LNI 192.168.203.56@tcp232 [ 1739.755591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1739.807328] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1740.690386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1740.741631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1740.793162] LNet: Removed LNI 192.168.203.56@tcp232 [ 1740.818734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1740.863943] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1741.678934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1741.714590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1741.745559] LNet: Removed LNI 192.168.203.56@tcp232 [ 1741.760457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1741.795312] LNet: Added LNI 192.168.203.56@tcp232 [8/256/0/180] [ 1742.527750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1742.574887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1742.610293] LNet: Removed LNI 192.168.203.56@tcp232 [ 1742.632310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1748.670729] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:54:45 (1788846885) [ 1749.053587] LNet: 88010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.058715] LNet: 88010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1749.541337] Key type .llcrypt unregistered [ 1749.543587] Key type ._llcrypt unregistered [ 1755.360354] Key type ._llcrypt registered [ 1755.362844] Key type .llcrypt registered [ 1755.582664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1755.600113] alg: No test for adler32 (adler32-zlib) [ 1756.545616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1756.573570] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1763.817408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1771.554799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.593878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1771.612754] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1773.281620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1773.328514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1773.378583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1773.420428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1773.475965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_interrupt -r 2 [ 1773.525541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_interrupt -r 2 [ 1773.568455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_interrupt -r 2 [ 1773.606328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_interrupt -r 2 [ 1773.649919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1773.689986] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1773.696845] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1773.953885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1773.977616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1774.015591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1774.037087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1774.059212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.093674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1774.128160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1774.157629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1774.173659] LNet: 88670:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.192386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1774.226047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1774.260891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1774.291736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1774.339182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1774.387882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1774.892470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1774.932496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1774.982715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1775.030467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1775.097636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_dropped -r 2 [ 1775.161469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_dropped -r 2 [ 1775.248764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_dropped -r 2 [ 1775.309666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_dropped -r 2 [ 1775.385441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1775.434295] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1775.799601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1775.846916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1775.924451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1775.977414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1776.038372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1776.109371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1776.156288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1776.239888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1776.286538] LNet: 88790:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.325507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1776.392700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1776.436143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1776.475692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1776.540406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1776.600368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1777.084490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1777.138228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1777.169305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1777.197991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1777.224863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_aborted -r 2 [ 1777.259906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_aborted -r 2 [ 1777.298229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_aborted -r 2 [ 1777.324720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_aborted -r 2 [ 1777.350704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1777.366390] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1777.583173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1777.607859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1777.634603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1777.659672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1777.683370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1777.708830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1777.733496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1777.769393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1777.789912] LNet: 88911:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1777.808550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1777.833619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1777.863046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1777.896677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1777.922682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1777.957903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1778.461092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1778.508493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1778.542390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1778.586244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1778.633211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_no_route -r 2 [ 1778.673645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_no_route -r 2 [ 1778.709272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_no_route -r 2 [ 1778.738444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_no_route -r 2 [ 1778.775391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1779.020819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1779.059819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1779.116245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1779.171307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1779.229179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1779.273308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1779.311194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1779.370214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1779.419596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1779.468074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1779.510734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1779.604642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1779.643646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1779.700462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1780.306408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1780.369413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1780.414055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1780.458468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1780.487350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e local_timeout -r 2 [ 1780.521262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e local_timeout -r 2 [ 1780.560775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e local_timeout -r 2 [ 1780.601888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e local_timeout -r 2 [ 1780.644780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1780.676303] LNet: There was an unexpected network error while writing to 192.168.203.156: rc = -22 [ 1780.682653] LNet: Skipped 1 previous similar message [ 1780.976780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1781.022168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1781.073917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1781.125849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1781.176801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1781.236639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1781.283540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1781.331551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1781.361255] LNet: 89151:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.374309] LNet: 89151:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1781.406380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1781.443489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1781.484461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1781.531273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1781.551841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1781.601528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1782.198179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1782.242744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1782.295170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1782.376081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1782.432812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp -m GET -e remote_dropped -r 2 [ 1782.494566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.56@tcp1 -d 192.168.203.156@tcp1 -m GET -e remote_dropped -r 2 [ 1782.538882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp -m GET -e remote_dropped -r 2 [ 1782.572884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.56@tcp1 -d 192.168.203.56@tcp1 -m GET -e remote_dropped -r 2 [ 1782.630104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.56@tcp 192.168.203.156@tcp [ 1782.666551] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 1782.924958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1782.957729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1782.997568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1783.059500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1783.106120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1783.138812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1783.179201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1783.204971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1783.245061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1783.292616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1783.324151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1783.358215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1783.402925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1783.452376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1789.021849] LNet: 89346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.037957] LNetError: 89346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.049285] LNet: Removed LNI 192.168.203.56@tcp [ 1789.061150] LNet: Removed LNI 192.168.203.56@tcp1 [ 1789.650522] Key type .llcrypt unregistered [ 1789.655952] Key type ._llcrypt unregistered [ 1808.807605] Key type ._llcrypt registered [ 1808.813788] Key type .llcrypt registered [ 1810.617850] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:55:46 (1788846946) [ 1811.022823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.032990] alg: No test for adler32 (adler32-zlib) [ 1811.944649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.033890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.078378] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1813.043369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1813.098391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.56@tcp [ 1813.128486] LNetError: 90322:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1813.260158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.56@tcp [ 1813.420325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1820.674382] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:55:57 (1788846957) [ 1821.251623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.292269] LNet: 90913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.306240] LNetError: 90913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.340932] LNet: Removed LNI 192.168.203.56@tcp [ 1821.394804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.519204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.585494] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1822.686367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1822.723291] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 1824.245563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.56@tcp [ 1824.267657] LNetError: 91180:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.56@tcp to recovery [ 1824.543429] LNet: 1 local NIs in recovery (showing 1): 192.168.203.56@tcp [ 1824.546411] LNet: There was an unexpected network error while writing to 192.168.203.56: rc = -22 [ 1825.550754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.56@tcp [ 1833.011696] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:56:10 (1788846970) [ 1835.008190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg356-server [ 1836.644985] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:56:13 (1788846973) [ 1838.156573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg356-server [ 1839.904807] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:56:16 (1788846976) [ 1841.978443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg356-server [ 1844.050310] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:56:20 (1788846980) [ 1844.197855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.230761] LNet: 92796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.241413] LNetError: 92796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.250428] LNet: Removed LNI 192.168.203.56@tcp [ 1844.267993] LNet: Removed LNI 192.168.203.56@tcp1 [ 1844.306894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.397627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.429290] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1845.312349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1845.413276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1845.463822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1845.663248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1845.735478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1845.868577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1852.957479] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:56:29 (1788846989) [ 1853.065282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.094688] LNet: 93527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.101228] LNetError: 93527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.108925] LNetError: 93527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1853.123821] LNet: Removed LNI 192.168.203.56@tcp [ 1853.156381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.206702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1853.224094] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1860.339591] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:56:37 (1788846997) [ 1860.530823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.567593] LNet: 94234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.574339] LNetError: 94234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.618314] LNet: Removed LNI 192.168.203.56@tcp [ 1860.667813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.731501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.768054] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1868.217762] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:56:44 (1788847004) [ 1870.114542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1872.020543] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:56:48 (1788847008) [ 1872.757391] LNet: 95271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.766579] LNetError: 95271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.783272] LNet: Removed LNI 192.168.203.56@tcp [ 1873.522971] Key type .llcrypt unregistered [ 1873.526309] Key type ._llcrypt unregistered [ 1879.985525] Key type ._llcrypt registered [ 1879.990403] Key type .llcrypt registered [ 1880.216504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.233655] alg: No test for adler32 (adler32-zlib) [ 1881.165539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1881.195796] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1889.233985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 1894.243598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.156@tcp [ 1894.527742] LNet: 95686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.534058] LNetError: 95686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.553333] LNet: Removed LNI 192.168.203.56@tcp [ 1894.902684] Key type .llcrypt unregistered [ 1894.905599] Key type ._llcrypt unregistered [ 1906.414027] Key type ._llcrypt registered [ 1906.418914] Key type .llcrypt registered [ 1908.295729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1910.388842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1912.606434] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:57:29 (1788847049) [ 1914.305782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1915.942681] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:57:33 (1788847053) [ 1918.020853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg356-server [ 1920.190210] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:57:36 (1788847056) [ 1922.643467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg356-server [ 1924.913794] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:57:41 (1788847061) [ 1926.581784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1926.650531] alg: No test for adler32 (adler32-zlib) [ 1927.604897] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 1927.880371] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1929.668368] Key type lgssc registered [ 1930.776839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.269872] Key type lgssc unregistered [ 1949.634696] LNet: 98856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.643215] LNetError: 98856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.666514] LNet: Removed LNI 192.168.203.56@tcp [ 1950.295642] Key type .llcrypt unregistered [ 1950.297258] Key type ._llcrypt unregistered [ 1967.017177] Key type ._llcrypt registered [ 1967.021921] Key type .llcrypt registered [ 1968.565163] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:58:25 (1788847105) [ 1969.263813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.275739] alg: No test for adler32 (adler32-zlib) [ 1970.209730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.278134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.328330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.374941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.420361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1976.439481] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:58:33 (1788847113) [ 1976.681111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.726310] LNet: 100268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.775683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.867592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.898136] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1977.943833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.966550] LNet: 100412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.972712] LNetError: 100412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.993915] LNet: Removed LNI 192.168.203.56@tcp [ 1978.010654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.062728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1978.089886] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1978.717105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.739987] LNetError: 100440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.753291] LNet: Removed LNI 192.168.203.56@tcp [ 1978.770290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.845481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1978.898470] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1985.535347] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:58:42 (1788847122) [ 1985.666881] LNet: 101026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.674095] LNet: 101026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1985.684790] LNetError: 101026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.721046] LNet: Removed LNI 192.168.203.56@tcp [ 1986.332678] Key type .llcrypt unregistered [ 1986.337684] Key type ._llcrypt unregistered [ 1993.083943] Key type ._llcrypt registered [ 1993.089114] Key type .llcrypt registered [ 1993.350821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1993.360476] alg: No test for adler32 (adler32-zlib) [ 1994.410331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.488419] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 1994.492664] LNet: *** cfs_fail_loc=e003, val=100*** [ 1995.345992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 1995.365649] LNet: *** cfs_fail_loc=e003, val=100*** [ 1995.367966] LNet: Skipped 2 previous similar messages [ 1995.374853] Ping reply received with mapping: [ 1995.377460] found metadata reply NI[192.168.203.56@tcp] [ 1995.380710] 38 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1995.384454] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1995.388094] 5a 5a 5a 5a 5a 5a 5a 5a [ 1995.609993] LNet: 101425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.632702] LNetError: 101425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.673681] LNet: Removed LNI 192.168.203.56@tcp [ 1996.525595] Key type .llcrypt unregistered [ 1996.528873] Key type ._llcrypt unregistered [ 2007.885747] Key type ._llcrypt registered [ 2007.889647] Key type .llcrypt registered [ 2009.345820] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:59:06 (1788847146) [ 2009.825118] Key type .llcrypt unregistered [ 2009.827417] Key type ._llcrypt unregistered [ 2016.016949] Key type ._llcrypt registered [ 2016.019303] Key type .llcrypt registered [ 2016.282059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2016.306926] alg: No test for adler32 (adler32-zlib) [ 2017.420680] LNetError: 102461:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2017.831410] Key type .llcrypt unregistered [ 2017.833283] Key type ._llcrypt unregistered [ 2023.958238] Key type ._llcrypt registered [ 2023.967549] Key type .llcrypt registered [ 2025.419427] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:59:22 (1788847162) [ 2026.683796] Key type .llcrypt unregistered [ 2026.687156] Key type ._llcrypt unregistered [ 2034.692253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2034.701691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2036.312626] Key type ._llcrypt registered [ 2036.314318] Key type .llcrypt registered [ 2036.515128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2036.526042] alg: No test for adler32 (adler32-zlib) [ 2037.422447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.493453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2048.056205] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:59:44 (1788847184) [ 2048.344929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.366601] LNet: 104226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.379542] LNetError: 104226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2048.406339] LNet: Removed LNI 10.1.2.3@tcp [ 2048.430263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.055760] Key type .llcrypt unregistered [ 2049.060830] Key type ._llcrypt unregistered [ 2056.304336] Key type ._llcrypt registered [ 2056.312018] Key type .llcrypt registered [ 2056.722557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2056.757927] alg: No test for adler32 (adler32-zlib) [ 2057.824591] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2057.943614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2058.133596] LNet: 104512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.145306] LNetError: 104512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.176442] LNet: Removed LNI 192.168.203.56@tcp [ 2058.919574] Key type .llcrypt unregistered [ 2058.920982] Key type ._llcrypt unregistered [ 2070.966571] Key type ._llcrypt registered [ 2070.969549] Key type .llcrypt registered [ 2072.247934] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:00:09 (1788847209) [ 2073.017717] Key type .llcrypt unregistered [ 2073.021539] Key type ._llcrypt unregistered [ 2079.293916] Key type ._llcrypt registered [ 2079.300323] Key type .llcrypt registered [ 2079.518685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.537447] alg: No test for adler32 (adler32-zlib) [ 2086.841552] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:00:23 (1788847223) [ 2086.955251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.006543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.129753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.174399] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2087.958839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2087.999515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2088.032867] LNet: Removed LNI 192.168.203.56@tcp [ 2088.245508] LNet: 106252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.559837] Key type .llcrypt unregistered [ 2088.567251] Key type ._llcrypt unregistered [ 2099.570832] Key type ._llcrypt registered [ 2099.572336] Key type .llcrypt registered [ 2100.954665] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:00:37 (1788847237) [ 2101.275611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2101.298350] alg: No test for adler32 (adler32-zlib) [ 2102.240979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.313389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.333988] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2103.222917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2103.283231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.56@tcp [ 2103.311290] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2103.318338] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.325542] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2103, 0, 0 [ 2103.336967] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2103.341299] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2103.346459] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.350476] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2103, 0, 0 [ 2103.359808] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2103.361962] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2103.371195] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.374098] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2103, 0, 2153 [ 2103.380953] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2103.384186] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2103.387271] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2103.391988] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2103, 0, 2153 [ 2103.396347] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2110.389320] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:00:47 (1788847247) [ 2111.371624] LNet: 107788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.375396] LNetError: 107788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.391646] LNet: Removed LNI 192.168.203.56@tcp [ 2111.846360] Key type .llcrypt unregistered [ 2111.849765] Key type ._llcrypt unregistered [ 2119.486677] Key type ._llcrypt registered [ 2119.491204] Key type .llcrypt registered [ 2119.722647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.753730] alg: No test for adler32 (adler32-zlib) [ 2120.702328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2120.746995] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2129.850935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2139.120736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.168622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2139.197609] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 2139.954409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2140.105901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2140.113190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2140.117066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2142.390191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2142.430662] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2152.580604] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:01:29 (1788847289) [ 2153.084909] LNet: 109309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.106644] LNetError: 109309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.122732] LNet: Removed LNI 192.168.203.56@tcp [ 2154.143177] LNet: Removed LNI 192.168.203.56@tcp1 [ 2154.152214] LNetError: 109309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2154.158031] LNetError: 109309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2154.183696] LNet: Removed LNI 10.1.2.3@tcp99 [ 2154.804518] Key type .llcrypt unregistered [ 2154.808654] Key type ._llcrypt unregistered [ 2161.788695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2161.793792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2164.617457] Key type ._llcrypt registered [ 2164.621084] Key type .llcrypt registered [ 2164.837142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2164.859830] alg: No test for adler32 (adler32-zlib) [ 2165.801105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.890858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.920322] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2166.651448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2166.672920] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2168.602902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2168.643679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2168.687632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2168.706507] LNetError: 110051:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2168.738300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2168.781431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2169.086329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2169.138870] LNet: 110071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.144617] LNetError: 110071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2169.151544] LNet: Removed LNI 10.1.2.3@tcp [ 2170.216672] LNet: Removed LNI 192.168.203.56@tcp [ 2170.807347] Key type .llcrypt unregistered [ 2170.811245] Key type ._llcrypt unregistered [ 2183.312439] Key type ._llcrypt registered [ 2183.314883] Key type .llcrypt registered [ 2184.625927] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:02:01 (1788847321) [ 2185.141996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.159699] alg: No test for adler32 (adler32-zlib) [ 2186.067427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.120810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.145342] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2186.846566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2186.966406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg356-client@tcp [ 2193.904600] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:02:10 (1788847330) [ 2194.201544] LNet: 111600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.213062] LNetError: 111600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.235170] LNet: Removed LNI 192.168.203.56@tcp [ 2194.902578] Key type .llcrypt unregistered [ 2194.909540] Key type ._llcrypt unregistered [ 2201.735871] Key type ._llcrypt registered [ 2201.741092] Key type .llcrypt registered [ 2201.895422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.910938] alg: No test for adler32 (adler32-zlib) [ 2202.818211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2202.849514] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2202.869416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2208.763231] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:02:25 (1788847345) [ 2210.426920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2212.090590] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:02:29 (1788847349) [ 2213.768729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2215.912449] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:02:32 (1788847352) [ 2216.021849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.079409] LNet: 113013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.082796] LNetError: 113013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2216.100158] LNet: Removed LNI 192.168.203.56@tcp [ 2216.153978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.288847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.327876] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2217.029734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2217.091697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2223.488973] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:02:40 (1788847360) [ 2223.630541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2223.655198] LNet: 113728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.671745] LNetError: 113728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.703231] LNet: Removed LNI 192.168.203.56@tcp [ 2223.718270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.762651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2223.809922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2224.336409] Key type .llcrypt unregistered [ 2224.340121] Key type ._llcrypt unregistered [ 2236.964269] Key type ._llcrypt registered [ 2236.967989] Key type .llcrypt registered [ 2238.728857] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:02:55 (1788847375) [ 2239.309598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.320877] alg: No test for adler32 (adler32-zlib) [ 2240.414939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.501840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.543272] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2241.425344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2241.509643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2248.613570] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:03:05 (1788847385) [ 2248.771158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.810331] LNet: 115276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.819235] LNetError: 115276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.858171] LNet: Removed LNI 192.168.203.56@tcp [ 2248.883187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.965912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2249.025932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2255.514890] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:03:12 (1788847392) [ 2255.643483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2255.687397] LNet: 115858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.739667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.827884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2256.253954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2256.313820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2256.398468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2256.906567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2256.973941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2257.091885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.111742] LNet: 115896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.157915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.264686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2257.876379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2257.958865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2258.017860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2258.794029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2258.844418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2266.953376] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:03:23 (1788847403) [ 2267.118273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.147242] LNet: 116494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.181305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.249992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.276037] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2268.124796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2268.161388] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2269.447614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2269.536570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2269.846478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.864504] LNetError: 116777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.883382] LNet: Removed LNI 192.168.203.56@tcp [ 2270.894575] LNetError: 116777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.905067] LNet: Removed LNI 192.168.203.56@tcp2 [ 2270.967544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.077904] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2271.088509] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2271.094642] LNet: 116783:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.796151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2272.026263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.049144] LNet: 116820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.056265] LNet: 116820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2273.119328] LNet: Removed LNI 192.168.203.56@tcp [ 2273.122815] LNetError: 116820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.134989] LNetError: 116820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2273.162769] LNet: Removed LNI 192.168.203.56@tcp2 [ 2273.205775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.268236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.303571] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2274.040298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2274.070965] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2275.029381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2275.087961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2275.173706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2275.457439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.511283] LNet: Removed LNI 192.168.203.56@tcp [ 2277.542528] LNet: Removed LNI 192.168.203.56@tcp2 [ 2277.611523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.684947] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2277.694610] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2277.699629] LNet: 117111:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2278.189684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2278.397989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2278.426878] LNetError: 117148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.436870] LNetError: 117148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2278.451077] LNet: Removed LNI 192.168.203.56@tcp [ 2279.461719] LNet: Removed LNI 192.168.203.56@tcp2 [ 2279.499865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.571293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2279.603075] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2280.307113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2280.324516] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2281.095855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2281.187743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2281.457180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.495603] LNet: 117428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.502497] LNet: 117428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2282.530374] LNet: Removed LNI 192.168.203.56@tcp [ 2282.541059] LNet: Removed LNI 192.168.203.56@tcp2 [ 2282.573702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.658367] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2282.666648] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2282.676777] LNet: 117434:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.155383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2283.332811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.448124] LNet: Removed LNI 192.168.203.56@tcp [ 2284.466777] LNet: Removed LNI 192.168.203.56@tcp2 [ 2284.507305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.586913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2284.613078] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2285.483200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2285.522242] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2286.465112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2286.512470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2286.588715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2286.827542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.852263] LNetError: 117756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.857914] LNetError: 117756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2286.863842] LNet: Removed LNI 192.168.203.56@tcp [ 2287.912632] LNet: Removed LNI 192.168.203.56@tcp2 [ 2287.939155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.011191] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2288.023832] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2288.031196] LNet: 117762:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.744936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2288.959869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.982838] LNet: Removed LNI 192.168.203.56@tcp [ 2290.030920] LNet: Removed LNI 192.168.203.56@tcp2 [ 2290.058402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.157515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.184191] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2291.063170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2291.109303] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2292.149986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2292.206962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2292.329067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2292.778718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.855826] LNet: Removed LNI 192.168.203.56@tcp [ 2293.874217] LNet: Removed LNI 192.168.203.56@tcp2 [ 2293.912743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.981986] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2293.988853] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2293.995453] LNet: 118102:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2294.709754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2295.140761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.223164] LNet: Removed LNI 192.168.203.56@tcp [ 2296.243449] LNet: Removed LNI 192.168.203.56@tcp2 [ 2296.285649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.342850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.370356] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2297.082755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2297.112159] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2298.006794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2298.054485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2298.100874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2298.160669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2298.536894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.556463] LNet: 118457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.563304] LNet: 118457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2299.615146] LNet: Removed LNI 192.168.203.56@tcp [ 2300.655528] LNet: Removed LNI 192.168.203.56@tcp2 [ 2300.692105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.762724] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2300.768397] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2301.189323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2308.693174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2310.393449] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:04:07 (1788847447) [ 2310.674120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.706443] LNetError: 119224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.714274] LNetError: 119224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2310.722705] LNet: Removed LNI 192.168.203.56@tcp [ 2311.787221] LNet: Removed LNI 192.168.203.56@tcp2 [ 2311.812950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.086993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.131349] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2312.830693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2312.854283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2313.552777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.574696] LNet: Removed LNI 10.1.2.3@tcp [ 2313.587837] LNet: Removed LNI 192.168.203.56@tcp [ 2313.608476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.656628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2313.675753] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2313.680448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2314.162440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2314.195783] LNet: Removed LNI 192.168.203.56@tcp [ 2314.215209] LNet: Removed LNI 10.1.2.3@tcp [ 2321.214328] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 02:04:17 (1788847457) [ 2321.329472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.386817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2321.478016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2321.499939] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2322.348039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2322.408716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.451646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.500430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.539680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.567344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.591803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.622934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.657253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.705561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.747258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.56@tcp [ 2322.875786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2329.159919] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:04:26 (1788847466) [ 2329.434895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.535759] LNet: Removed LNI 192.168.203.56@tcp [ 2330.558810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.652648] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2330.752255] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2330.761389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2330.769916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2333.175436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.204563] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2334.058695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2334.077598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2336.492355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2336.545815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2336.609403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.56@tcp -m PUT -e local_timeout -r 1 [ 2336.670982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2336.738531] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2336.744603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2337.015813] LNetError: 120903:0:(lib-move.c:4801:lnet_parse()) 192.168.203.56@tcp, src 192.168.203.56@tcp: Dropping PUT (error -108 looking up sender) [ 2338.015652] LNet: 121363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.024800] LNet: 121363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2338.037957] LNet: Removed LNI 10.1.2.3@tcp [ 2339.111277] LNet: Removed LNI 192.168.203.56@tcp [ 2339.558294] Key type .llcrypt unregistered [ 2339.565192] Key type ._llcrypt unregistered [ 2351.230428] Key type ._llcrypt registered [ 2351.233100] Key type .llcrypt registered [ 2352.549041] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:04:49 (1788847489) [ 2352.891917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2352.910850] alg: No test for adler32 (adler32-zlib) [ 2353.860217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.312855] LNet: 122213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.993377] Key type .llcrypt unregistered [ 2354.998000] Key type ._llcrypt unregistered [ 2361.187309] Key type ._llcrypt registered [ 2361.195614] Key type .llcrypt registered [ 2361.491669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2361.519951] alg: No test for adler32 (adler32-zlib) [ 2362.507521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2362.580097] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2369.440556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2369.655590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2370.458365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2370.726444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2370.766825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2370.785493] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2370.811655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2370.866952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2370.921268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2370.965812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2371.021345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2371.060408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2371.088580] LNet: 122661:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2371.120952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2371.157902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2371.197249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2371.238765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2371.268225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2371.295121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2371.989402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2372.221311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2372.250600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2372.321328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2372.381227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2372.448374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2372.511519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2372.584983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2372.669460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2372.702441] LNet: 122778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2372.732409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2372.797630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2372.860190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2372.911758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2372.946945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2372.983543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2373.675898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2373.900949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2373.939666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2373.975532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2374.004199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2374.043254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2374.067935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2374.109519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2374.142737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2374.158924] LNet: 122895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.176485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2374.213312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2374.278883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2374.314168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2374.358828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2374.408298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2375.021475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2375.240655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2375.278701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2375.314470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2375.355119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2375.397299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.438817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.491325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.524336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2375.561485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2375.589625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2375.639170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2375.677321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2375.719447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2375.757525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.130842] LNet: 123058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2376.150279] LNetError: 123058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.169507] LNet: Removed LNI 192.168.203.56@tcp [ 2376.578979] Key type .llcrypt unregistered [ 2376.582027] Key type ._llcrypt unregistered [ 2395.844347] Key type ._llcrypt registered [ 2395.848150] Key type .llcrypt registered [ 2397.220780] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:05:34 (1788847534) [ 2397.792976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.810585] alg: No test for adler32 (adler32-zlib) [ 2398.818681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.333695] LNet: 123944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.110592] Key type .llcrypt unregistered [ 2400.113772] Key type ._llcrypt unregistered [ 2406.607374] Key type ._llcrypt registered [ 2406.608887] Key type .llcrypt registered [ 2406.805085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2406.820838] alg: No test for adler32 (adler32-zlib) [ 2407.810104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2407.855396] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2414.936207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.156@tcp [ 2423.217000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.264289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2423.295889] LNet: Added LNI 192.168.203.56@tcp1 [8/256/0/180] [ 2424.181766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2424.780318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.156@tcp [ 2425.058380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2425.087256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2425.129876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.176411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.213502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.243783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.249268] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.156@tcp [ 2425.284907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.327029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.365665] LNet: 124594:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2425.397397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.461392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.511604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.574987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.632890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.690325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2426.448204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2426.750372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2426.777613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2426.813243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2426.846535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2426.882178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2426.906605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2426.948000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2426.982929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2427.009773] LNet: 124711:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.039391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2427.086404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2427.126494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2427.159756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2427.200982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2427.241541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.907976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.156@tcp [ 2428.132872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2428.160307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2428.184056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2428.235114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2428.258995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2428.280670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2428.315969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2428.362766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2428.386426] LNet: 124828:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2428.407907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2428.437701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2428.474346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2428.521531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2428.587080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2428.638485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.291665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2429.555224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2429.592458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2429.624553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2429.661756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2429.685773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2429.726580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2429.764296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2429.788881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2429.835989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2429.874822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2429.928347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2429.974735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2430.038387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2430.097726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2435.523892] LNet: 125027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.531364] LNetError: 125027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.540308] LNet: Removed LNI 192.168.203.56@tcp [ 2435.548445] LNet: Removed LNI 192.168.203.56@tcp1 [ 2436.136352] Key type .llcrypt unregistered [ 2436.139370] Key type ._llcrypt unregistered [ 2452.454267] Key type ._llcrypt registered [ 2452.457319] Key type .llcrypt registered [ 2453.913568] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:06:30 (1788847590) [ 2454.610451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.643463] alg: No test for adler32 (adler32-zlib) [ 2455.584758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.669947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2455.704995] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2456.328928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2456.354722] LNet: Added LNI 192.168.203.56@tcp2 [8/256/0/180] [ 2457.222755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.56@tcp [ 2457.295498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2458.335302] LNet: Removed LNI 192.168.203.56@tcp [ 2464.756874] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:06:41 (1788847601) [ 2466.385199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2468.398545] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:06:45 (1788847605) [ 2476.304753] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:06:53 (1788847613) [ 2483.673939] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:07:00 (1788847620) [ 2491.301264] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:07:08 (1788847628) [ 2498.200535] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:07:15 (1788847635) [ 2505.851680] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:07:22 (1788847642) [ 2514.065335] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:07:30 (1788847650) [ 2522.396358] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:07:39 (1788847659) [ 2530.152737] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:07:46 (1788847666) [ 2537.504337] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:07:54 (1788847674) [ 2544.208174] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:08:00 (1788847680) [ 2550.923951] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:08:08 (1788847688) [ 2557.948924] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:08:15 (1788847695) [ 2565.249581] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:08:22 (1788847702) [ 2573.650878] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:08:30 (1788847710) [ 2581.370292] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:08:38 (1788847718) [ 2589.659727] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:08:46 (1788847726) [ 2597.371555] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:08:54 (1788847734) [ 2597.638377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2597.662988] LNet: 136721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2597.671959] LNetError: 136721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.700652] LNet: Removed LNI 192.168.203.56@tcp2 [ 2598.736520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2598.913964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2598.942302] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2600.095360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2600.132948] LNet: 136863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2600.142921] LNetError: 136863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2600.163368] LNet: Removed LNI 192.168.203.56@tcp [ 2600.194748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2600.265505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2600.284243] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2607.071770] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:09:04 (1788847744) [ 2614.415436] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:09:11 (1788847751) [ 2621.967723] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:09:19 (1788847759) [ 2629.329204] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:09:26 (1788847766) [ 2637.533057] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:09:33 (1788847773) [ 2646.859916] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:09:43 (1788847783) [ 2657.288672] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:09:53 (1788847793) [ 2666.181816] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:10:03 (1788847803) [ 2674.461984] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:10:11 (1788847811) [ 2682.105484] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:10:19 (1788847819) [ 2690.965947] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:10:27 (1788847827) [ 2700.114734] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:10:36 (1788847836) [ 2708.643985] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2449 sec ========== 02:10:45 (1788847845) [ 2708.941625] LNet: 144333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2708.948838] LNetError: 144333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2708.965086] LNet: Removed LNI 192.168.203.56@tcp [ 2709.671643] Key type .llcrypt unregistered [ 2709.676227] Key type ._llcrypt unregistered [ 2722.691720] Key type ._llcrypt registered [ 2722.692869] Key type .llcrypt registered [ 2722.923634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2722.935891] alg: No test for adler32 (adler32-zlib) [ 2724.095127] Lustre: Lustre: Build Version: 2.17.57_108_g4a8ba26 [ 2724.343062] LNet: Added LNI 192.168.203.56@tcp [8/256/0/180] [ 2726.055213] Key type lgssc registered [ 2727.268494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2798.848701] Lustre: Mounted lustre-client - version 2.17.57_108_g4a8ba26 [ 2804.286098] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2814.228465] Lustre: Unmounted lustre-client [ 2840.252898] Key type lgssc unregistered [ 2840.682519] LNet: 147284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2840.697038] LNetError: 147284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2840.717615] LNet: Removed LNI 192.168.203.56@tcp [ 2841.723157] Key type .llcrypt unregistered [ 2841.728324] Key type ._llcrypt unregistered