[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 417591018 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002337] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006351] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009163] LSM: Security Framework initializing [ 0.010045] Yama: becoming mindful. [ 0.011051] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.020753] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024666] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027085] *** VALIDATE tmpfs *** [ 0.028447] *** VALIDATE proc *** [ 0.029203] *** VALIDATE cgroup *** [ 0.030011] *** VALIDATE cgroup2 *** [ 0.031262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035029] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.038606] debug: unmapping init [mem 0xffffffff9ea59000-0xffffffff9ea60fff] [ 0.040173] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041587] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042026] ... version: 2 [ 0.043009] ... bit width: 48 [ 0.043934] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045010] ... max period: 00007fffffffffff [ 0.046007] ... fixed-purpose events: 3 [ 0.046934] ... event mask: 000000070000000f [ 0.047263] rcu: Hierarchical SRCU implementation. [ 0.049160] smp: Bringing up secondary CPUs ... [ 0.050570] x86: Booting SMP configuration: [ 0.051020] .... node #0, CPUs: #1 #2 #3 [ 0.056022] smp: Brought up 1 node, 4 CPUs [ 0.058027] smpboot: Max logical packages: 1 [ 0.059015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140944] node 0 deferred pages initialised in 80ms [ 0.143368] devtmpfs: initialized [ 0.144284] x86/mm: Memory block size: 128MB [ 0.147792] gcov: version magic: 0x41383552 [ 0.151117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156344] pinctrl core: initialized pinctrl subsystem [ 0.158187] [ 0.158626] ************************************************************* [ 0.160012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161009] ** ** [ 0.163010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.164046] ** ** [ 0.166012] ** This means that this kernel is built to expose internal ** [ 0.167008] ** IOMMU data structures, which may compromise security on ** [ 0.169009] ** your system. ** [ 0.170011] ** ** [ 0.172015] ** If you see this message and you are not debugging the ** [ 0.174012] ** kernel, report this immediately to your vendor! ** [ 0.176044] ** ** [ 0.178014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180065] ************************************************************* [ 0.183169] NET: Registered protocol family 16 [ 0.185442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.195009] cpuidle: using governor menu [ 0.196460] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198782] PCI: Using configuration type 1 for base access [ 0.200135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.211027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214045] cryptd: max_cpu_qlen set to 1000 [ 0.216208] ACPI: Added _OSI(Module Device) [ 0.217008] ACPI: Added _OSI(Processor Device) [ 0.218014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218909] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222349] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.227582] ACPI: Interpreter enabled [ 0.229052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.230009] ACPI: Using IOAPIC for interrupt routing [ 0.231099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.234425] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.246201] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.249071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.251027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.254121] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.258458] acpiphp: Slot [2] registered [ 0.259133] acpiphp: Slot [5] registered [ 0.261118] acpiphp: Slot [6] registered [ 0.262085] acpiphp: Slot [3] registered [ 0.263117] acpiphp: Slot [4] registered [ 0.264173] acpiphp: Slot [7] registered [ 0.266095] acpiphp: Slot [8] registered [ 0.267127] acpiphp: Slot [9] registered [ 0.268096] acpiphp: Slot [10] registered [ 0.270081] acpiphp: Slot [11] registered [ 0.271082] acpiphp: Slot [12] registered [ 0.272069] acpiphp: Slot [13] registered [ 0.273078] acpiphp: Slot [14] registered [ 0.274010] acpiphp: Slot [15] registered [ 0.275070] acpiphp: Slot [16] registered [ 0.276076] acpiphp: Slot [17] registered [ 0.277157] acpiphp: Slot [18] registered [ 0.279054] acpiphp: Slot [19] registered [ 0.280105] acpiphp: Slot [20] registered [ 0.281087] acpiphp: Slot [21] registered [ 0.282084] acpiphp: Slot [22] registered [ 0.284098] acpiphp: Slot [23] registered [ 0.285094] acpiphp: Slot [24] registered [ 0.286065] acpiphp: Slot [25] registered [ 0.286961] acpiphp: Slot [26] registered [ 0.288050] acpiphp: Slot [27] registered [ 0.289100] acpiphp: Slot [28] registered [ 0.291088] acpiphp: Slot [29] registered [ 0.292095] acpiphp: Slot [30] registered [ 0.293100] acpiphp: Slot [31] registered [ 0.294085] PCI host bridge to bus 0000:00 [ 0.296020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.298021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310325] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.315488] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322485] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.326055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336609] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339016] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344548] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348018] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.355013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.359015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.363204] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.387014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.395985] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.400014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.405015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.422023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.433421] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.435279] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.436279] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.438479] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.440183] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.443102] iommu: Default domain type: Passthrough [ 0.444423] SCSI subsystem initialized [ 0.445228] ACPI: bus type USB registered [ 0.447097] usbcore: registered new interface driver usbfs [ 0.448059] usbcore: registered new interface driver hub [ 0.449064] usbcore: registered new device driver usb [ 0.451154] pps_core: LinuxPPS API ver. 1 registered [ 0.452007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.454066] PTP clock support registered [ 0.456062] EDAC MC: Ver: 3.0.0 [ 0.457125] PCI: Using ACPI for IRQ routing [ 0.458000] NetLabel: Initializing [ 0.458007] NetLabel: domain hash size = 128 [ 0.459008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.460087] NetLabel: unlabeled traffic allowed by default [ 0.462269] vgaarb: loaded [ 0.463310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.464008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.470000] clocksource: Switched to clocksource kvm-clock [ 0.571770] VFS: Disk quotas dquot_6.6.0 [ 0.572708] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.575376] *** VALIDATE ramfs *** [ 0.576538] *** VALIDATE hugetlbfs *** [ 0.578051] pnp: PnP ACPI init [ 0.580153] pnp: PnP ACPI: found 6 devices [ 0.599715] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.602253] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.603631] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.605225] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.606692] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.608468] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.610824] NET: Registered protocol family 2 [ 0.613151] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.617503] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.620270] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.625524] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.628403] TCP: Hash tables configured (established 65536 bind 65536) [ 0.630863] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.633419] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.635834] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.638345] NET: Registered protocol family 1 [ 0.641452] RPC: Registered named UNIX socket transport module. [ 0.643507] RPC: Registered udp transport module. [ 0.645068] RPC: Registered tcp transport module. [ 0.646802] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.649254] NET: Registered protocol family 44 [ 0.650517] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.652130] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.653822] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.655623] PCI: CLS 0 bytes, default 64 [ 0.657299] Unpacking initramfs... [ 2.063645] debug: unmapping init [mem 0xffff8b5d7cc64000-0xffff8b5d7ffcffff] [ 2.067099] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.069410] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.071888] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.580912] Initialise system trusted keyrings [ 2.582099] Key type blacklist registered [ 2.583288] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.595786] zbud: loaded [ 2.600250] *** VALIDATE nfs *** [ 2.601630] *** VALIDATE nfs4 *** [ 2.603618] pstore: using deflate compression [ 2.609666] Platform Keyring initialized [ 2.736829] NET: Registered protocol family 38 [ 2.738879] Key type asymmetric registered [ 2.740144] Asymmetric key parser 'x509' registered [ 2.742339] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.745607] io scheduler mq-deadline registered [ 2.747631] io scheduler kyber registered [ 2.749528] io scheduler bfq registered [ 2.751899] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.755267] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.758535] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.761765] ACPI: Power Button [PWRF] [ 2.768196] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.775357] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.784938] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.812954] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.841962] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.847238] Non-volatile memory driver v1.3 [ 2.848603] Linux agpgart interface v0.103 [ 2.882181] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.884352] vda: detected capacity change from 0 to 74838016 [ 2.908491] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.911844] vdb: detected capacity change from 0 to 1073741824 [ 2.921453] libphy: Fixed MDIO Bus: probed [ 2.933098] usbcore: registered new interface driver usbserial_generic [ 2.936078] usbserial: USB Serial support registered for generic [ 2.938378] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.943559] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.945497] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.948228] mousedev: PS/2 mouse device common for all mice [ 2.952210] rtc_cmos 00:05: RTC can wake from S4 [ 2.955575] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.955956] rtc_cmos 00:05: registered as rtc0 [ 2.961408] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.961946] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.964533] intel_pstate: CPU model not supported [ 2.969219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.972097] hid: raw HID events driver (C) Jiri Kosina [ 2.973472] usbcore: registered new interface driver usbhid [ 2.974692] usbhid: USB HID core driver [ 2.976383] drop_monitor: Initializing network drop monitor service [ 2.978826] Initializing XFRM netlink socket [ 2.980696] NET: Registered protocol family 10 [ 2.982419] Segment Routing with IPv6 [ 2.983324] NET: Registered protocol family 17 [ 2.984803] mpls_gso: MPLS GSO support [ 2.989944] RAS: Correctable Errors collector initialized. [ 2.992370] AVX version of gcm_enc/dec engaged. [ 2.994318] AES CTR mode by8 optimization enabled [ 3.069655] sched_clock: Marking stable (3069615713, 0)->(3862914781, -793299068) [ 3.073299] registered taskstats version 1 [ 3.075990] Loading compiled-in X.509 certificates [ 3.077950] zswap: loaded using pool lzo/zbud [ 3.104225] Key type big_key registered [ 3.116683] Key type encrypted registered [ 3.118443] ima: No TPM chip found, activating TPM-bypass! [ 3.120574] ima: Allocated hash algorithm: sha1 [ 3.122462] ima: No architecture policies found [ 3.124402] evm: Initialising EVM extended attributes: [ 3.126442] evm: security.selinux [ 3.127391] evm: security.ima [ 3.128629] evm: security.capability [ 3.129799] evm: HMAC attrs: 0x1 [ 3.132053] rtc_cmos 00:05: setting system clock to 2026-08-24 08:29:16 UTC (1787560156) [ 3.137890] debug: unmapping init [mem 0xffffffff9fa03000-0xffffffff9fbfffff] [ 3.140399] debug: unmapping init [mem 0xffffffff9e782000-0xffffffff9ea58fff] [ 3.147149] Write protecting the kernel read-only data: 28672k [ 3.150772] debug: unmapping init [mem 0xffffffff9ce03000-0xffffffff9cffffff] [ 3.153769] debug: unmapping init [mem 0xffffffff9d714000-0xffffffff9d7fffff] [ 3.186966] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.194954] systemd[1]: Detected virtualization kvm. [ 3.196395] systemd[1]: Detected architecture x86-64. [ 3.198222] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.226574] systemd[1]: No hostname configured. [ 3.228217] systemd[1]: Set hostname to . [ 3.229957] random: systemd: uninitialized urandom read (16 bytes read) [ 3.231937] systemd[1]: Initializing machine ID from random generator. [ 3.361982] random: systemd: uninitialized urandom read (16 bytes read) [ 3.364360] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.368314] random: systemd: uninitialized urandom read (16 bytes read) [ 3.370282] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.373263] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.887965] device-mapper: uevent: version 1.0.3 [ 3.890150] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.538491] virtio_net virtio0 ens2: renamed from eth0 [ 4.590369] scsi host0: ata_piix [ 4.631148] scsi host1: ata_piix [ 4.632549] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.634201] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.312760] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.634180] random: crng init done [ 9.637557] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.986941] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.092776] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.303529] SELinux: Disabled at runtime. [ 11.358744] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.364465] systemd[1]: Detected virtualization kvm. [ 11.365556] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.832301] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.835594] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.840262] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.844188] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.848191] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.855239] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.858819] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File[ 12.010427] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.189577] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.477744] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.482938] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.548257] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.562260] EDAC sbridge: Ver: 1.1.2 [ 13.575949] Key type dns_resolver registered [ 13.894968] NFS: Registering the id_resolver key type [ 13.897518] Key type id_resolver registered [ 13.899381] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg255-client login: [ 50.514273] libcfs: loading out-of-tree module taints kernel. [ 50.581929] Key type ._llcrypt registered [ 50.584694] Key type .llcrypt registered [ 51.457854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 51.488824] alg: No test for adler32 (adler32-zlib) [ 52.959294] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 53.826830] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 55.615182] Key type lgssc registered [ 57.504288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 132.129023] hrtimer: interrupt took 3088295 ns [ 220.405746] Lustre: Mounted lustre-client - version 2.17.57_82_g4029ffa [ 225.515423] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.547830] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing check_logdir /tmp/testlogs/ [ 246.242152] Lustre: lustre-OST0000-osc-ffff8b5dd44a1800: disconnect after 23s idle [ 248.961745] Lustre: DEBUG MARKER: oleg255-client.virtnet: executing yml_node [ 252.594834] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 255.121501] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 257.596180] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 259.415675] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 04:33:31 EDT 2026 [ 274.693456] Lustre: Unmounted lustre-client [ 302.280962] Key type lgssc unregistered [ 302.495885] LNet: 8439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.500578] LNetError: 8439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.522925] LNet: Removed LNI 192.168.202.55@tcp [ 303.242169] Key type .llcrypt unregistered [ 303.244781] Key type ._llcrypt unregistered [ 327.813763] Key type ._llcrypt registered [ 327.815751] Key type .llcrypt registered [ 328.026503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 328.041686] alg: No test for adler32 (adler32-zlib) [ 328.988626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 329.067933] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 329.095351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 332.800349] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 332.955141] LNet: 9327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.964741] LNetError: 9327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.978753] LNet: Removed LNI 192.168.202.55@tcp [ 333.587428] Key type .llcrypt unregistered [ 333.589186] Key type ._llcrypt unregistered [ 339.974782] Key type ._llcrypt registered [ 339.976508] Key type .llcrypt registered [ 341.589862] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:34:53 (1787560493) [ 342.008625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 342.023490] alg: No test for adler32 (adler32-zlib) [ 343.129909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.327713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.373796] LNet: 9743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.435783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.547573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.615236] LNet: 9750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 351.385753] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:35:03 (1787560503) [ 351.750039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.833666] LNet: 10327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.871386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.996151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 361.681721] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:35:13 (1787560513) [ 361.810303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.845400] LNet: 10906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.894171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.087704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 372.331546] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:35:23 (1787560523) [ 372.436941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.514608] LNet: 11491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.558292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.696959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 381.524478] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:35:33 (1787560533) [ 381.654795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.689345] LNet: 12079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.721486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.859620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 382.343112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 388.819882] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:35:41 (1787560541) [ 388.945903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.023848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.184112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 389.848634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.465820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 398.282501] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:35:50 (1787560550) [ 398.411799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.443983] LNet: 13266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.453077] LNet: 13266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 398.483025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.574133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 406.513680] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:35:58 (1787560558) [ 406.836645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.905834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.987804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 407.025822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.505694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 407.549177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 407.773706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 407.834690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.167845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 408.204747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 408.550886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 408.582289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 408.899581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 408.937753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.201785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 409.226552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.473461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 409.519448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.896789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 409.929048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 410.276452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 410.338871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.709513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.748240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 411.296900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 411.345480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.628275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.664432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 421.146436] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:36:12 (1787560572) [ 421.391173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.439625] LNet: 14534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.453577] LNet: 14534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 421.511753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.716783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 421.782964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 430.650526] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:36:22 (1787560582) [ 431.008449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.067460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.175406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 431.221482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 439.380996] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:36:30 (1787560590) [ 439.712681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.755660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.908693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 439.952996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 448.186801] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:36:39 (1787560599) [ 448.256200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.301917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.380779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 448.419572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 455.277299] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:36:47 (1787560607) [ 455.562769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.588144] LNet: 16887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.592609] LNet: 16887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 455.614747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.724843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 455.811528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 462.708275] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:36:54 (1787560614) [ 462.820427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.882501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.017483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 463.067716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 470.211410] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:37:01 (1787560621) [ 470.324200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.418525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.566360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.142779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.847317] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:37:10 (1787560630) [ 479.297449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.389450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.716913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.099805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.461369] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:37:21 (1787560641) [ 489.847834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.911318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.029939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.260401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.371738] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:37:30 (1787560650) [ 498.592910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.641925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.142994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 499.412258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.714893] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:37:38 (1787560658) [ 506.841383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.891429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.963889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.041839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 515.140632] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:37:46 (1787560666) [ 515.413510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.466096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.540960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.582877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.976709] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:37:55 (1787560675) [ 523.358425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.377186] LNet: 21743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.382819] LNet: 21743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 523.427623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.506902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 530.459813] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:38:02 (1787560682) [ 530.737737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.789155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.843910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 538.829687] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:38:10 (1787560690) [ 538.906695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.962309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.082964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 539.704291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 547.547598] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:38:19 (1787560699) [ 548.038280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.078692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.206975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 548.276778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 555.946764] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:38:27 (1787560707) [ 556.156720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.215526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.344915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 556.388950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 563.393457] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:38:35 (1787560715) [ 563.630445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.719682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.811611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 563.867648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 571.167623] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:38:43 (1787560723) [ 571.602221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.623719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.671838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.696994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 579.886196] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:38:51 (1787560731) [ 580.051356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.131223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.223442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 580.322622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 580.347824] LNetError: 25861:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 580.394155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 580.436301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 589.167631] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:39:00 (1787560740) [ 589.322350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.432098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.640457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9ONpbcokm [ 589.756686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9ONpbcokm [ 589.829060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.N9ONpbcokm [ 599.525986] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:39:10 (1787560750) [ 599.603898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.651466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.755549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 599.798432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 599.847248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 608.952385] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 04:39:20 (1787560760) [ 609.172760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.267152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.383427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.412107] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 612.051853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 613.435190] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 615.682564] Key type lgssc registered [ 617.737162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.012643] LNetError: 27624:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.55@tcp99 has no available nets [ 626.683370] Key type lgssc unregistered [ 627.040948] LNetError: 28737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.066902] LNet: Removed LNI 192.168.202.55@tcp [ 627.775526] Key type .llcrypt unregistered [ 627.779123] Key type ._llcrypt unregistered [ 635.331498] Key type ._llcrypt registered [ 635.334082] Key type .llcrypt registered [ 636.809069] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:39:48 (1787560788) [ 637.582296] Key type .llcrypt unregistered [ 637.587120] Key type ._llcrypt unregistered [ 643.454981] Key type ._llcrypt registered [ 643.459160] Key type .llcrypt registered [ 643.684951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.704363] alg: No test for adler32 (adler32-zlib) [ 644.703208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.786883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 644.815552] LNet: 29673:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 644.825097] LNet: 29673:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 644.837272] LNet: 29673:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 644.866716] LNet: Message forwarding will be enabled in 2s [ 647.967161] LNet: Message forwarding enabled [ 648.505974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.555885] LNet: Message forwarding will stop in 2s to 6s [ 652.065554] LNet: Message forwarding disabled [ 653.187361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 653.217435] LNet: 29771:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 653.224741] LNet: 29771:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 653.235320] LNet: 29771:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 653.261718] LNet: Message forwarding will be enabled in 2s [ 653.501302] LNet: 29784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.370405] Key type .llcrypt unregistered [ 654.373266] Key type ._llcrypt unregistered [ 668.156711] Key type ._llcrypt registered [ 668.164540] Key type .llcrypt registered [ 669.387952] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:40:21 (1787560821) [ 670.046448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.064097] alg: No test for adler32 (adler32-zlib) [ 671.024674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.106152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 677.736098] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:40:29 (1787560829) [ 677.803307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.828587] LNet: 31182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.860648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.944401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 677.984444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 678.012568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 678.061639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 678.114159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 678.150653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 678.186575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 678.240839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 678.279460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 678.337741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 678.399456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 678.455763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 678.506252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 678.539812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 678.573282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 678.628193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 678.687880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 678.738589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 678.763433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 678.777534] LNetError: 31233:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 685.072264] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:40:37 (1787560837) [ 685.292715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.338729] LNet: 31802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.381391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.551269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.193838] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:40:47 (1787560847) [ 698.037919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg255-server [ 700.358109] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:40:51 (1787560851) [ 700.553819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.599852] LNet: 32733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.658569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.781212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.831116] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 701.667826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[56-60]@tcp [ 701.691425] LNet: 32869:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 701.763131] LNetError: 32740:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.56@tcp is being used as a gateway but routing feature is not turned on [ 701.776365] LNetError: 32740:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.56@tcp has gone from up to down [ 708.970405] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:41:01 (1787560861) [ 711.167278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg255-server [ 713.459253] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:41:05 (1787560865) [ 713.548219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.569241] LNet: 33800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.587585] LNetError: 33800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.622210] LNet: Removed LNI 192.168.202.55@tcp [ 713.679577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.792703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.832348] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 714.785968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[56-61/2]@tcp [ 714.826076] LNet: 33925:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 714.843338] LNet: 33925:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 714.868597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[56-61/2]@tcp [ 714.875353] LNetError: 33811:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.58@tcp is being used as a gateway but routing feature is not turned on [ 714.902810] LNetError: 33811:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 714.914963] LNetError: 33811:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.58@tcp has gone from up to down [ 714.929499] LNetError: 33811:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 722.833674] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:41:14 (1787560874) [ 722.965957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.003312] LNet: 34510:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.016478] LNetError: 34510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.082791] LNet: Removed LNI 192.168.202.55@tcp [ 723.130923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.274866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 723.340985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.385507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.437468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.486234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.530661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.605087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 723.686345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.715267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.776586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.849191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.875442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.917388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 724.012460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.061869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.136601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.246696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.281812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.365337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 724.461447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.491774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.568778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 724.723637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.778713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.851633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 724.985364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.095879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.190514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 725.287237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.341186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.425427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 725.570841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.633839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.770819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 733.073727] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:41:25 (1787560885) [ 735.235228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg255-server [ 737.271387] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:41:29 (1787560889) [ 739.026207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg255-server [ 740.727718] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:41:32 (1787560892) [ 740.968468] LNet: 35918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.973716] LNet: 35918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 741.566852] Key type .llcrypt unregistered [ 741.570517] Key type ._llcrypt unregistered [ 747.675513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.944894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.959219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.971285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 749.454832] Key type ._llcrypt registered [ 749.457449] Key type .llcrypt registered [ 749.698901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.729782] alg: No test for adler32 (adler32-zlib) [ 750.704387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.757508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.776150] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 751.990484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 752.031196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 754.876135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 754.911196] LNet: Removed LNI 10.1.2.3@tcp [ 754.985487] LNet: 36635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.004858] LNetError: 36635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.032523] LNet: Removed LNI 192.168.202.55@tcp [ 755.635755] Key type .llcrypt unregistered [ 755.639761] Key type ._llcrypt unregistered [ 768.781467] Key type ._llcrypt registered [ 768.784411] Key type .llcrypt registered [ 770.396765] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:42:01 (1787560921) [ 770.827536] Key type .llcrypt unregistered [ 770.829844] Key type ._llcrypt unregistered [ 777.109257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.252118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.256866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.261127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 780.056892] Key type ._llcrypt registered [ 780.060546] Key type .llcrypt registered [ 780.318528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.335544] alg: No test for adler32 (adler32-zlib) [ 781.223404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.311294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.351868] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 781.925729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.942576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.631078] LNet: 38132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.650463] LNetError: 38132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.662428] LNet: Removed LNI 10.1.2.3@tcp [ 784.676190] LNet: Removed LNI 192.168.202.55@tcp [ 785.460993] Key type .llcrypt unregistered [ 785.462793] Key type ._llcrypt unregistered [ 797.860119] Key type ._llcrypt registered [ 797.867986] Key type .llcrypt registered [ 799.156675] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:42:31 (1787560951) [ 799.738493] Key type .llcrypt unregistered [ 799.742497] Key type ._llcrypt unregistered [ 805.584885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 805.697136] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 805.709480] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.722127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.347428] Key type ._llcrypt registered [ 808.349449] Key type .llcrypt registered [ 808.482864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.502511] alg: No test for adler32 (adler32-zlib) [ 809.475878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.561085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 809.605872] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 811.639967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.653873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.707826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 813.745199] LNet: Removed LNI 10.1.2.3@tcp [ 813.759427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 813.808616] LNet: Removed LNI 10.1.2.31@tcp [ 813.841614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 813.884572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.806335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 814.824250] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 815.593701] LNet: 40057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.596814] LNetError: 40057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 815.600212] LNet: Removed LNI 10.1.2.31@tcp [ 815.606655] LNet: Removed LNI 10.1.2.3@tcp [ 816.051718] Key type .llcrypt unregistered [ 816.055159] Key type ._llcrypt unregistered [ 829.410139] Key type ._llcrypt registered [ 829.412019] Key type .llcrypt registered [ 830.689474] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:43:02 (1787560982) [ 831.022168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.036920] alg: No test for adler32 (adler32-zlib) [ 832.130476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.193924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.218829] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 832.930983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.961435] LNet: 41014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.971950] LNetError: 41014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.990497] LNet: Removed LNI 192.168.202.55@tcp [ 833.017545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.118297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.55@tcp [ 833.140232] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 833.943784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.55@tcp [ 833.962807] LNet: Removed LNI 192.168.202.55@tcp [ 834.029339] LNet: 41149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.784873] Key type .llcrypt unregistered [ 834.799480] Key type ._llcrypt unregistered [ 847.913290] Key type ._llcrypt registered [ 847.915387] Key type .llcrypt registered [ 849.200384] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:43:21 (1787561001) [ 851.361130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg255-server [ 853.260224] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:43:25 (1787561005) [ 853.916471] Key type .llcrypt unregistered [ 853.917966] Key type ._llcrypt unregistered [ 860.735683] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.831403] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.842438] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.853469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.716769] Key type ._llcrypt registered [ 863.719681] Key type .llcrypt registered [ 863.837646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.849544] alg: No test for adler32 (adler32-zlib) [ 864.720250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.753561] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 865.237209] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 866.964653] LNet: 42992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.977520] LNetError: 42992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.982992] LNet: Removed LNI 192.168.202.55@tcp1 [ 866.993847] LNet: Removed LNI 10.1.2.3@tcp2 [ 867.560629] Key type .llcrypt unregistered [ 867.562882] Key type ._llcrypt unregistered [ 880.420024] Key type ._llcrypt registered [ 880.423059] Key type .llcrypt registered [ 882.002596] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:43:53 (1787561033) [ 882.413696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.426291] alg: No test for adler32 (adler32-zlib) [ 883.419609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.484953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.510983] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 884.307406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.353505] LNet: 43966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.371470] LNetError: 43966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.398637] LNet: Removed LNI 192.168.202.55@tcp [ 884.421852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.453319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 884.463938] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 892.332973] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:44:03 (1787561043) [ 892.486372] LNet: 44554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.492573] LNetError: 44554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.516867] LNet: Removed LNI 192.168.202.55@tcp [ 893.258666] Key type .llcrypt unregistered [ 893.261927] Key type ._llcrypt unregistered [ 900.546039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.678320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 900.689735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 900.704966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 903.339424] Key type ._llcrypt registered [ 903.341962] Key type .llcrypt registered [ 903.476100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.499268] alg: No test for adler32 (adler32-zlib) [ 904.468182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.520551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.538640] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 905.440780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 905.471168] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.599153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.642512] LNet: 45306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.653281] LNetError: 45306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.662262] LNet: Removed LNI 10.1.2.3@tcp [ 907.672978] LNet: Removed LNI 192.168.202.55@tcp [ 907.689081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.733711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 907.749048] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 907.752280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.241678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.279558] LNet: 45334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.285480] LNetError: 45334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.299246] LNetError: 45334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 908.307479] LNet: Removed LNI 10.1.2.3@tcp [ 908.327248] LNet: Removed LNI 192.168.202.55@tcp [ 908.372982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.473399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 908.549124] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 908.564520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.062739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 909.082417] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 910.329854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.365297] LNet: 45496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.371555] LNetError: 45496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.377840] LNetError: 45496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 910.384387] LNet: Removed LNI 10.1.2.3@tcp [ 910.386971] LNet: Removed LNI 192.168.202.55@tcp [ 910.405523] LNet: Removed LNI 192.168.202.55@tcp2 [ 910.462050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.560628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 910.599211] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 910.609605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.626236] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 918.644768] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:44:30 (1787561070) [ 918.976313] LNet: 46093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.989430] LNetError: 46093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.996367] LNetError: 46093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 919.002384] LNet: Removed LNI 10.1.2.3@tcp [ 919.003488] LNet: Removed LNI 192.168.202.55@tcp [ 919.017617] LNet: Removed LNI 192.168.202.55@tcp2 [ 919.889801] Key type .llcrypt unregistered [ 919.891743] Key type ._llcrypt unregistered [ 926.682615] Key type ._llcrypt registered [ 926.685421] Key type .llcrypt registered [ 926.977567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 926.993398] alg: No test for adler32 (adler32-zlib) [ 928.010300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.124203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.152940] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 929.151895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.164132] LNet: 46492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.168152] LNetError: 46492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.178809] LNet: Removed LNI 192.168.202.55@tcp [ 929.208861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.256355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.277117] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 929.608407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.641964] LNet: Removed LNI 192.168.202.55@tcp [ 929.659111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.738397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 929.765560] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 930.647420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 930.673948] LNet: 46552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.679852] LNet: 46552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 930.686854] LNetError: 46552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.693592] LNetError: 46552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 930.707819] LNet: Removed LNI 192.168.202.55@tcp [ 930.731433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.822846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 930.859932] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 931.430174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 931.571380] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 931.580301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 931.590553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.126198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.160771] LNet: 46631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.170307] LNetError: 46631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.181226] LNet: Removed LNI 192.168.202.55@tcp [ 934.198981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.279737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 934.304649] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 941.110579] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:44:53 (1787561093) [ 941.257091] LNet: 47222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.268334] LNetError: 47222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.290154] LNet: Removed LNI 192.168.202.55@tcp [ 942.044855] Key type .llcrypt unregistered [ 942.048159] Key type ._llcrypt unregistered [ 948.763111] Key type ._llcrypt registered [ 948.765870] Key type .llcrypt registered [ 949.052833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.095816] alg: No test for adler32 (adler32-zlib) [ 950.018249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.076128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.109841] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 951.019806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.034367] LNet: 47629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.037668] LNetError: 47629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.052420] LNet: Removed LNI 192.168.202.55@tcp [ 951.070234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.159855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 951.197173] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 958.268940] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:45:10 (1787561110) [ 958.379723] LNet: 48213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.389869] LNetError: 48213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.419582] LNet: Removed LNI 192.168.202.55@tcp [ 959.189744] Key type .llcrypt unregistered [ 959.193321] Key type ._llcrypt unregistered [ 965.282789] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.524971] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.543083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.553403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.330599] Key type ._llcrypt registered [ 968.332329] Key type .llcrypt registered [ 968.753289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.798170] alg: No test for adler32 (adler32-zlib) [ 969.887737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.989891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.044247] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 971.241638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.264251] LNet: 48675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.270930] LNetError: 48675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.288693] LNet: Removed LNI 192.168.202.55@tcp [ 971.309988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.410712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 971.461156] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 979.873561] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:45:31 (1787561131) [ 980.292396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.309609] LNet: 49273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.315156] LNetError: 49273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.337392] LNet: Removed LNI 192.168.202.55@tcp [ 980.364542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.499566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 987.631669] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:45:39 (1787561139) [ 988.182310] LNet: 49856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.068562] Key type .llcrypt unregistered [ 989.076582] Key type ._llcrypt unregistered [ 997.235461] Key type ._llcrypt registered [ 997.242381] Key type .llcrypt registered [ 997.602814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.630705] alg: No test for adler32 (adler32-zlib) [ 998.776814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.885436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.922835] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1000.572718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.638394] LNet: 50258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.647875] LNetError: 50258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.663805] LNet: Removed LNI 192.168.202.55@tcp [ 1000.695700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.792845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1000.834041] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1010.240641] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:46:02 (1787561162) [ 1010.454421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.471454] LNet: 50848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.478180] LNetError: 50848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.501627] LNet: Removed LNI 192.168.202.55@tcp [ 1010.548075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.598121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1018.069507] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:46:09 (1787561169) [ 1020.253029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1022.639551] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:46:14 (1787561174) [ 1025.019463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1027.078983] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:46:18 (1787561178) [ 1029.375528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1031.475559] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:46:23 (1787561183) [ 1033.847624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1036.123376] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:46:27 (1787561187) [ 1037.635509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1039.642719] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:46:31 (1787561191) [ 1039.787942] LNet: 52898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.526613] Key type .llcrypt unregistered [ 1040.529289] Key type ._llcrypt unregistered [ 1046.931922] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.085030] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1047.087473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1047.090222] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.752924] Key type ._llcrypt registered [ 1049.755183] Key type .llcrypt registered [ 1050.041032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.062870] alg: No test for adler32 (adler32-zlib) [ 1050.990302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.041454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1051.060988] LNet: Added LNI 192.168.202.55@tcp [8/512/0/180] [ 1051.066706] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1051.068460] LNet: 53229:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1051.173832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.196536] LNet: 53248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.204724] LNetError: 53248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1051.212802] LNet: Removed LNI 10.1.2.3@tcp [ 1051.215909] LNetError: 53248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.232581] LNet: Removed LNI 192.168.202.55@tcp [ 1051.261476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.353603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1051.382182] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1051.489568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.523039] LNet: 53272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.541211] LNetError: 53272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.569315] LNet: Removed LNI 192.168.202.55@tcp [ 1051.604933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.699740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1051.740966] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1051.752968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1051.758666] LNet: 53280:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1051.930256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.971252] LNet: 53298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.980497] LNetError: 53298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1051.987161] LNet: Removed LNI 10.1.2.3@tcp [ 1051.989519] LNetError: 53298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.013154] LNet: Removed LNI 192.168.202.55@tcp [ 1052.049467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.206643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1052.265962] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1052.275655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1052.277485] LNet: 53306:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1052.538331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.577219] LNet: 53332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.590418] LNetError: 53332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1052.601560] LNet: Removed LNI 10.1.2.3@tcp [ 1052.607314] LNetError: 53332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.629839] LNet: Removed LNI 192.168.202.55@tcp [ 1052.673572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.814501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1052.859150] LNet: Added LNI 192.168.202.55@tcp [8/256/0/182] [ 1052.874795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1053.010801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.042264] LNet: 53360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.055080] LNetError: 53360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1053.064344] LNet: Removed LNI 10.1.2.3@tcp [ 1053.065904] LNetError: 53360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.106270] LNet: Removed LNI 192.168.202.55@tcp [ 1053.164090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.299098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1053.363973] LNet: Added LNI 192.168.202.55@tcp [8/256/0/182] [ 1053.374972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1053.381556] LNet: 53368:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1053.498483] LNet: Removed LNI 10.1.2.3@tcp [ 1053.525084] LNet: Removed LNI 192.168.202.55@tcp [ 1054.447517] Key type .llcrypt unregistered [ 1054.449373] Key type ._llcrypt unregistered [ 1067.302397] Key type ._llcrypt registered [ 1067.303746] Key type .llcrypt registered [ 1068.621794] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 04:47:00 (1787561220) [ 1070.467373] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1072.668878] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:47:04 (1787561224) [ 1073.127382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1073.168388] alg: No test for adler32 (adler32-zlib) [ 1074.193324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.287963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1074.304719] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1075.126715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.55 [ 1075.144509] LNet: 54636:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1075.154691] LNet: Removed LNI 192.168.202.55@tcp [ 1081.925098] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:47:13 (1787561233) [ 1082.640556] LNet: 55206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.650653] LNetError: 55206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.685604] LNet: Removed LNI 192.168.202.55@tcp [ 1083.690539] Key type .llcrypt unregistered [ 1083.692563] Key type ._llcrypt unregistered [ 1092.958297] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1093.277197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1096.269288] Key type ._llcrypt registered [ 1096.275748] Key type .llcrypt registered [ 1096.583557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.608239] alg: No test for adler32 (adler32-zlib) [ 1098.037310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1105.764648] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:47:37 (1787561257) [ 1105.898256] LNet: 56116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.905389] LNetError: 56116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1105.922410] LNet: Removed LNI 10.1.2.3@tcp [ 1106.737518] Key type .llcrypt unregistered [ 1106.738946] Key type ._llcrypt unregistered [ 1113.878555] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1114.192557] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1114.200791] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1114.209365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1117.110872] Key type ._llcrypt registered [ 1117.114859] Key type .llcrypt registered [ 1117.307547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1117.327099] alg: No test for adler32 (adler32-zlib) [ 1118.247428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1125.364363] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:47:57 (1787561277) [ 1126.720425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1128.613196] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:48:00 (1787561280) [ 1128.761110] LNet: 57306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.778695] LNetError: 57306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.806441] LNet: Removed LNI 10.1.2.3@tcp [ 1129.527702] Key type .llcrypt unregistered [ 1129.529460] Key type ._llcrypt unregistered [ 1136.412030] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1136.691360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1136.697461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1136.710517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.505944] Key type ._llcrypt registered [ 1139.512556] Key type .llcrypt registered [ 1139.740781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.770287] alg: No test for adler32 (adler32-zlib) [ 1140.768379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1140.866772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.291607] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:48:22 (1787561302) [ 1151.777420] LNet: 58546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.782594] LNetError: 58546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.795502] LNet: Removed LNI 10.1.2.3@tcp [ 1152.585511] Key type .llcrypt unregistered [ 1152.588659] Key type ._llcrypt unregistered [ 1160.077233] Key type ._llcrypt registered [ 1160.080271] Key type .llcrypt registered [ 1160.420835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.443791] alg: No test for adler32 (adler32-zlib) [ 1161.470283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.515836] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1169.717348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1170.957592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1171.026955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1171.125969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1171.177990] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1171.549057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.592171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.654721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.774765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.809053] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1171.876715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.002165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.085068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.151558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.168204] LNet: 58989:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.195212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.223869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.256816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.291277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.322515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.345473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.377621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.358906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1173.408829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1173.462478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1173.486371] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1173.784785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.810434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.848095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.883871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.915887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.943538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.974189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.045775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.090966] LNet: 59099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.145757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.206423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.243914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.295479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.345411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.401760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.484246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.327941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1175.432165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1175.510281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1175.532657] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1176.048886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.116529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.164814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.198686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.242000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.343304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.394992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.462937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.491290] LNet: 59208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.509904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.558128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.649284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.745815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.815226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.897110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.012247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.043564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1178.104426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1178.182652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1178.221504] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1178.615601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.685880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.783556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.882861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.970929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.059280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.114451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.154666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.173726] LNet: 59318:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.215873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.260229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.320844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.378956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.455896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.504428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.584467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.625643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1180.666843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1180.697317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1181.111229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.227846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.298546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.362710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.387278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.461927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.481973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.497318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.515440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.543243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.598143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.620400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.744097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.815907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.923962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.703395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1182.749688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1182.832504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1182.898707] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1182.903756] LNet: Skipped 2 previous similar messages [ 1183.116881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.141438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.163909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.201917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.245875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.286918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.312149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.336858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.354391] LNet: 59536:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.359527] LNet: 59536:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1183.369918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.417900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.489158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.556633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.648363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.695136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.729925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.095480] LNet: 59582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.114381] LNetError: 59582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.147226] LNet: Removed LNI 192.168.202.55@tcp [ 1184.556525] Key type .llcrypt unregistered [ 1184.557536] Key type ._llcrypt unregistered [ 1206.422712] Key type ._llcrypt registered [ 1206.424523] Key type .llcrypt registered [ 1208.155122] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:49:19 (1787561359) [ 1209.482660] Key type .llcrypt unregistered [ 1209.491925] Key type ._llcrypt unregistered [ 1217.238195] Key type ._llcrypt registered [ 1217.239640] Key type .llcrypt registered [ 1217.630739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1217.668888] alg: No test for adler32 (adler32-zlib) [ 1218.750539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1218.800054] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1226.734303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1236.619586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.691917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.749158] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1238.331498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1238.365956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 1 [ 1238.410119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1238.480510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 1 [ 1238.516399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_interrupt -r 1 [ 1238.556837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 1 [ 1238.592439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_interrupt -r 1 [ 1238.642530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 1 [ 1238.683581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1238.703534] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1238.710025] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1238.998332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.028656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.056389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.092987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.130795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.168421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.203215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.240518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.261706] LNet: 61055:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.284683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.329852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.362365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.407361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.454962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.486853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.532400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.156901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1240.193556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 1 [ 1240.228656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1240.283126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 1 [ 1240.340105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_dropped -r 1 [ 1240.385994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 1 [ 1240.435665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_dropped -r 1 [ 1240.498533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 1 [ 1240.531448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1240.555277] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1240.562520] LNet: Skipped 2 previous similar messages [ 1240.786552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.831192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.888440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.924274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.955760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.983675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.019236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.049894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.074337] LNet: 61177:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.102158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.144953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.201545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.232335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.259881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.312619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.374903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.138362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1242.219938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 1 [ 1242.245683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1242.302843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 1 [ 1242.367451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_aborted -r 1 [ 1242.402672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 1 [ 1242.435933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_aborted -r 1 [ 1242.507481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 1 [ 1242.564610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1242.595342] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1242.604825] LNet: Skipped 2 previous similar messages [ 1242.856044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.892167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.922585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.952145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.979468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.021525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.045564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.069862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.085384] LNet: 61298:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.102393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.129819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.152421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.179158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.212439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.233462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.258171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.881673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1243.934380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 1 [ 1243.968580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1244.005596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 1 [ 1244.055439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_no_route -r 1 [ 1244.123562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 1 [ 1244.161555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_no_route -r 1 [ 1244.227205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 1 [ 1244.295117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1244.590626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.619596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.645741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.688840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.735792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.776695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.828838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.874572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.919205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.969277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.013542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.056721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.095500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.150432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.179375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.806233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1245.830537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 1 [ 1245.866656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1245.896535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 1 [ 1245.927435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_timeout -r 1 [ 1245.972169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 1 [ 1246.037371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_timeout -r 1 [ 1246.110406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 1 [ 1246.178828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1246.206104] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1246.216241] LNet: Skipped 5 previous similar messages [ 1246.403475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.431471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.460322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.482334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.512119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.535405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.562650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.591402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.606792] LNet: 61541:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.611765] LNet: 61541:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1246.628573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.663597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.687214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.706381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.729470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.758413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.781824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.310524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1247.347980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_error -r 1 [ 1247.389401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1247.428890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_error -r 1 [ 1247.466254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_error -r 1 [ 1247.496357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_error -r 1 [ 1247.546133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_error -r 1 [ 1247.599717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_error -r 1 [ 1247.668831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1247.976276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.011169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1248.052349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.082368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.126138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.156343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.183106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.208254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.230412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.254412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.286476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.317710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.344646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.370207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.393286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.116934] LNet: 61742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.121527] LNetError: 61742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.131167] LNet: Removed LNI 192.168.202.55@tcp [ 1254.182768] LNet: Removed LNI 192.168.202.55@tcp1 [ 1254.684744] Key type .llcrypt unregistered [ 1254.685897] Key type ._llcrypt unregistered [ 1275.083714] Key type ._llcrypt registered [ 1275.085786] Key type .llcrypt registered [ 1277.006114] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:50:28 (1787561428) [ 1278.895744] Key type .llcrypt unregistered [ 1278.916911] Key type ._llcrypt unregistered [ 1287.847797] Key type ._llcrypt registered [ 1287.849964] Key type .llcrypt registered [ 1288.116741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.151201] alg: No test for adler32 (adler32-zlib) [ 1289.139256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.209966] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1298.826926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1299.565245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1299.606691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1299.647094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1299.726923] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1300.159754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.243251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1300.325874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.379380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.462976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.511403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.559329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1300.599438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1300.629461] LNet: 62997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1300.647461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1300.701566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1300.765864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.801386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.831898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.880969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1300.925081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.925534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1301.990929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1302.029182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1302.041357] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1302.287951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.333418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.374888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.447960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.515394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.597603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.653486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.714115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.761322] LNet: 63107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.796572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.847179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.910537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.960993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.028063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.084631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1303.134430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.004636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1304.033258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1304.058539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1304.072861] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1304.242597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.267988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.304959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.331276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.380936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.418206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.462481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.525661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.561175] LNet: 63216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.581545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.642544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.684086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.744514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.793964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.847611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.885794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.466385] LNet: 63261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.489746] LNetError: 63261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.536818] LNet: Removed LNI 192.168.202.55@tcp [ 1307.043692] Key type .llcrypt unregistered [ 1307.045448] Key type ._llcrypt unregistered [ 1330.742089] Key type ._llcrypt registered [ 1330.745645] Key type .llcrypt registered [ 1332.693343] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:51:23 (1787561483) [ 1334.337289] Key type .llcrypt unregistered [ 1334.342507] Key type ._llcrypt unregistered [ 1341.771761] Key type ._llcrypt registered [ 1341.773101] Key type .llcrypt registered [ 1341.990991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1342.016206] alg: No test for adler32 (adler32-zlib) [ 1343.071615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1343.109980] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1351.445283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1363.894319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.044758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.105620] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1366.516850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1366.624737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 1 [ 1366.723577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1366.797756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 1 [ 1366.844629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_dropped -r 1 [ 1366.890616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 1 [ 1366.935962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_dropped -r 1 [ 1366.991539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 1 [ 1367.044606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1367.070547] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1367.078588] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1367.429573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.474675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.520520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.565161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.613309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.659985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.719991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.781497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.802434] LNet: 64736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.822486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.856733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.894297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.928767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.967754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.010310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.042247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1368.087522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.733509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1368.776258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_error -r 1 [ 1368.811316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1368.881873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_error -r 1 [ 1368.953274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_error -r 1 [ 1368.991605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_error -r 1 [ 1369.039602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_error -r 1 [ 1369.102522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_error -r 1 [ 1369.164478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1369.201402] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1369.206740] LNet: Skipped 2 previous similar messages [ 1369.634557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1369.665305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1369.714184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1369.755497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.803469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.852485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.917375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.987372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.013394] LNet: 64860:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.034541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.073224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.131224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.190402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.245482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.294505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1370.341468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1370.405904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1371.381145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1371.426510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_timeout -r 1 [ 1371.462767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1371.504481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_timeout -r 1 [ 1371.563764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_timeout -r 1 [ 1371.620567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_timeout -r 1 [ 1371.689316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_timeout -r 1 [ 1371.805473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_timeout -r 1 [ 1371.932084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.155@tcp [ 1371.992970] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1372.002361] LNet: Skipped 1 previous similar message [ 1372.482583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1372.555072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1372.606408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1372.669136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1372.696326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1372.734761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1372.773957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1372.812471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1372.834203] LNet: 64983:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1372.872288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1372.915462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1372.951274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.990277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.039291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.085537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1373.149279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1373.192126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.672074] LNet: 65066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1379.690563] LNetError: 65066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1379.700675] LNet: Removed LNI 192.168.202.55@tcp [ 1379.733692] LNet: Removed LNI 192.168.202.55@tcp1 [ 1380.259570] Key type .llcrypt unregistered [ 1380.261624] Key type ._llcrypt unregistered [ 1399.419344] Key type ._llcrypt registered [ 1399.421800] Key type .llcrypt registered [ 1400.718931] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:52:32 (1787561552) [ 1402.429911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1404.362585] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:52:36 (1787561556) [ 1405.427426] Key type .llcrypt unregistered [ 1405.429855] Key type ._llcrypt unregistered [ 1411.392621] Key type ._llcrypt registered [ 1411.394378] Key type .llcrypt registered [ 1411.535498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1411.551885] alg: No test for adler32 (adler32-zlib) [ 1412.537361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1412.568190] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1419.818826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1420.628784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1420.715436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1420.764903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1420.797476] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1421.093069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1421.132356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1421.179859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1421.216691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1421.260489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1421.299521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1421.330884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1421.358659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1421.386486] LNet: 66614:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1421.427145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1421.474504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1421.528596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.569935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1421.631627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.722973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1422.264393] LNet: 66657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.288169] LNetError: 66657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.315144] LNet: Removed LNI 192.168.202.55@tcp [ 1422.892387] Key type .llcrypt unregistered [ 1422.896512] Key type ._llcrypt unregistered [ 1443.976475] Key type ._llcrypt registered [ 1443.981373] Key type .llcrypt registered [ 1444.179645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1444.196399] alg: No test for adler32 (adler32-zlib) [ 1445.174324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1445.212518] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1453.158718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1462.534649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.588349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1462.640408] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1464.147625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1464.182585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e network_timeout -r 1 [ 1464.210752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1464.250385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e network_timeout -r 1 [ 1464.298585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e network_timeout -r 1 [ 1464.353247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e network_timeout -r 1 [ 1464.402659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e network_timeout -r 1 [ 1464.452501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e network_timeout -r 1 [ 1464.506774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1464.534408] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1464.672744] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1464.677837] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1464.840995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1464.877107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1464.902844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1464.936555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1464.969974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1464.997632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1465.058200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1465.115721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1465.134182] LNet: 67542:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1465.150587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1465.181684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1465.214725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.244414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1465.275586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1465.315513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1469.700279] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1470.231079] LNet: 67620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.245542] LNetError: 67620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.252546] LNet: Removed LNI 192.168.202.55@tcp [ 1470.265486] LNet: Removed LNI 192.168.202.55@tcp1 [ 1470.712770] Key type .llcrypt unregistered [ 1470.713861] Key type ._llcrypt unregistered [ 1488.429855] Key type ._llcrypt registered [ 1488.431485] Key type .llcrypt registered [ 1489.644634] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:54:01 (1787561641) [ 1490.086200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1490.107710] alg: No test for adler32 (adler32-zlib) [ 1491.022279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.062381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.077747] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1491.750302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.770834] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1492.681699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1492.749484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1492.796714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1492.828968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1492.863345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1492.896752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1492.926754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1492.949358] LNetError: 68721:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1493.087246] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1493.090023] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1495.138045] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1495.383354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1499.233717] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1500.104622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1500.175286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1500.304989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.345629] LNet: 68828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.354446] LNetError: 68828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.407183] LNet: Removed LNI 192.168.202.55@tcp [ 1501.410129] LNetError: 68828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.432188] LNet: Removed LNI 192.168.202.55@tcp1 [ 1501.480685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.567270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.596042] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1502.363420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.387525] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1503.336322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1503.373172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1503.447432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1503.521473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1503.579260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1503.626285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1503.662503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.692590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1503.717105] LNetError: 69101:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1504.613285] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1507.236726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1510.754443] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1510.764187] LNet: Skipped 1 previous similar message [ 1515.285615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1515.316090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1515.345446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.610763] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:54:33 (1787561673) [ 1521.807124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1521.832375] LNet: 69816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.839488] LNetError: 69816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.847604] LNet: Removed LNI 192.168.202.55@tcp [ 1522.864130] LNet: Removed LNI 192.168.202.55@tcp1 [ 1522.910145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.992242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1523.029384] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1523.702698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.726029] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1524.515620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1524.587239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1524.614837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1524.643415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1524.677128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1524.707684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1524.741616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 0 [ 1524.826557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.023693] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.55@tcp [ 1525.033440] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1525.046143] LNet: Skipped 1 previous similar message [ 1539.801849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1540.019815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1540.081826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.139735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 500 [ 1540.233732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1547.126840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1547.227425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.264826] LNet: 70303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.280722] LNetError: 70303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.285772] LNetError: 70303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1547.299871] LNet: Removed LNI 192.168.202.55@tcp [ 1547.319458] LNet: Removed LNI 192.168.202.55@tcp1 [ 1547.358297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.489502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.535598] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1548.338658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1548.381218] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1549.537275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1549.592692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.696595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1549.744044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1549.811787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1549.878218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1549.959744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1550.020752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.55@tcp --health 0 [ 1551.519333] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1551.522379] LNet: Skipped 3 previous similar messages [ 1552.564942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.001379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1562.034658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1562.076451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1569.976268] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:55:21 (1787561721) [ 1570.526198] LNet: 71348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.536501] LNetError: 71348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.551365] LNetError: 71348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.564814] LNet: Removed LNI 192.168.202.55@tcp [ 1570.593693] LNet: Removed LNI 192.168.202.55@tcp1 [ 1571.769078] Key type .llcrypt unregistered [ 1571.773084] Key type ._llcrypt unregistered [ 1580.806227] Key type ._llcrypt registered [ 1580.813713] Key type .llcrypt registered [ 1581.089348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1581.134441] alg: No test for adler32 (adler32-zlib) [ 1582.069502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1582.103920] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1592.203983] Lustre: DEBUG MARKER: Initial discovery [ 1592.255544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1594.314606] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1594.423398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1594.470559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1594.504695] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1594.544238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1594.610344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.155@tcp -r 1 -m GET -l 3 [ 1594.656736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.155@tcp -r 1 -m GET -e local_error [ 1594.703592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.155@tcp -r 1 -m PUT -l 6 [ 1595.425384] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1596.053835] Lustre: DEBUG MARKER: Force oleg255-client.virtnet to discover 192.168.202.155@tcp (in background) [ 1596.134747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.155@tcp [ 1597.572795] Lustre: DEBUG MARKER: Force oleg255-server to discover 192.168.202.55@tcp [ 1599.199640] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1600.240388] Lustre: DEBUG MARKER: Wait for 72107 [ 1601.901886] Lustre: DEBUG MARKER: Finished wait on 72107 [ 1601.985107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1602.031575] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1602.050499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1602.398605] LNet: 72569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.403945] LNetError: 72569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.432904] LNet: Removed LNI 192.168.202.55@tcp [ 1603.996789] Key type .llcrypt unregistered [ 1604.000472] Key type ._llcrypt unregistered [ 1620.787930] Key type ._llcrypt registered [ 1620.792021] Key type .llcrypt registered [ 1622.016858] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:56:13 (1787561773) [ 1622.524959] Key type .llcrypt unregistered [ 1622.529804] Key type ._llcrypt unregistered [ 1628.641493] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1628.893173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1628.902234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1628.917579] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1631.504973] Key type ._llcrypt registered [ 1631.507933] Key type .llcrypt registered [ 1631.718242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.754606] alg: No test for adler32 (adler32-zlib) [ 1632.724281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.812868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.856403] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1633.609832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1633.637690] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1643.489872] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:56:35 (1787561795) [ 1643.690153] LNet: 74651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.699212] LNetError: 74651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1643.703384] LNet: Removed LNI 10.1.2.3@tcp [ 1643.714048] LNet: Removed LNI 192.168.202.55@tcp [ 1644.567598] Key type .llcrypt unregistered [ 1644.569175] Key type ._llcrypt unregistered [ 1652.176160] Key type ._llcrypt registered [ 1652.181126] Key type .llcrypt registered [ 1652.403625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1652.431484] alg: No test for adler32 (adler32-zlib) [ 1653.349773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.400771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.423722] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1654.346606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1654.363077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1655.455866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.55@tcp [ 1656.496461] LNetError: Unexpected error -22 connecting to 192.168.202.55@tcp at host 192.168.202.55:988 [ 1662.515039] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:56:54 (1787561814) [ 1662.604772] LNet: 75780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.612802] LNetError: 75780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1662.624187] LNet: Removed LNI 10.1.2.3@tcp [ 1662.649446] LNet: Removed LNI 192.168.202.55@tcp [ 1663.560492] Key type .llcrypt unregistered [ 1663.562211] Key type ._llcrypt unregistered [ 1669.952976] Key type ._llcrypt registered [ 1669.959482] Key type .llcrypt registered [ 1670.146463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1670.186657] alg: No test for adler32 (adler32-zlib) [ 1671.158348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.218742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1671.246971] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1671.994321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1672.013032] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 1672.919658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.55@tcp1 --nid 192.168.202.55@tcp2 [ 1677.418301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp1 192.168.202.55@tcp2 [ 1677.442527] LNetError: 76629:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.55@tcp1 to 192.168.202.55@tcp2 - Route table may be misconfigured [ 1677.472544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp2 192.168.202.55@tcp1 [ 1684.690925] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:57:16 (1787561836) [ 1684.811684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.862711] LNet: 77196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.873645] LNetError: 77196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.885580] LNet: Removed LNI 192.168.202.55@tcp1 [ 1685.932303] LNet: Removed LNI 192.168.202.55@tcp2 [ 1685.989578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.147789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.201430] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1687.355580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1687.380074] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1688.448963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 1688.526372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.55@tcp [ 1688.561370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp1 192.168.202.55@tcp1 [ 1688.591174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.55@tcp -d 192.168.202.55@tcp -e network_timeout [ 1688.619602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -e network_timeout [ 1688.649947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -e network_timeout [ 1688.684598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -e network_timeout [ 1688.722176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1688.777606] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1688.827708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1688.994498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1689.088435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1689.124658] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1696.741543] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:57:28 (1787561848) [ 1696.917610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.951951] LNet: 78065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.959580] LNetError: 78065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.966386] LNetError: 78065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1696.970506] LNet: Removed LNI 192.168.202.55@tcp [ 1698.025974] LNet: Removed LNI 192.168.202.55@tcp1 [ 1698.113301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.304795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1698.687314] LNet: 78106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.385535] Key type .llcrypt unregistered [ 1699.398848] Key type ._llcrypt unregistered [ 1715.018647] Key type ._llcrypt registered [ 1715.021663] Key type .llcrypt registered [ 1716.369159] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:57:48 (1787561868) [ 1716.853788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.887340] alg: No test for adler32 (adler32-zlib) [ 1717.999426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.838472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1722.103640] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:57:53 (1787561873) [ 1722.189828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.225935] LNet: 79205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.259378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.382116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.472779] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1723.529753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.547628] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1724.881438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1724.936897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp1 [ 1724.991570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp1 [ 1732.757898] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:58:04 (1787561884) [ 1735.487879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg255-server [ 1737.229192] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:58:09 (1787561889) [ 1739.537922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg255-server [ 1741.510238] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:58:13 (1787561893) [ 1743.886443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg255-server [ 1746.026241] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:58:17 (1787561897) [ 1747.499745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg255-server [ 1750.073234] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:58:21 (1787561901) [ 1752.727357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg255-server [ 1754.837615] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:58:26 (1787561906) [ 1756.663879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg255-server [ 1759.019941] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:58:30 (1787561910) [ 1761.045911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg255-server [ 1763.168497] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:58:35 (1787561915) [ 1765.946292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg255-server [ 1768.779431] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:58:39 (1787561919) [ 1771.425910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg255-server [ 1773.533362] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:58:45 (1787561925) [ 1773.958551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.012566] LNet: 83093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.033022] LNetError: 83093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.063415] LNet: Removed LNI 192.168.202.55@tcp [ 1775.081854] LNet: Removed LNI 192.168.202.55@tcp1 [ 1775.187129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.325138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1775.377145] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1776.630310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1776.835366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1777.165915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.210660] LNet: 83240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.227135] LNetError: 83240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.250926] LNetError: 83240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1777.289541] LNet: Removed LNI 192.168.202.55@tcp [ 1777.324220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.405838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1777.450421] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1778.050230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1778.243516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.323070] LNetError: 83385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1778.368700] LNet: Removed LNI 192.168.202.55@tcp [ 1778.436447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.533366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.551986] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1779.717944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1779.831147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1780.071668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.113212] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1780.116930] LNet: 83533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1781.156759] LNet: Removed LNI 192.168.202.55@tcp [ 1781.214324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.255630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1781.284771] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1781.838933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1782.001604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.025486] LNetError: 83678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.028937] LNetError: 83678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1782.078617] LNet: Removed LNI 192.168.202.55@tcp [ 1782.105415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.199555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.249946] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1783.079816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1783.302504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1783.527418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.613373] LNet: Removed LNI 192.168.202.55@tcp [ 1784.665837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.779148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1784.826892] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1785.922533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1786.175926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.197415] LNet: 83970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.205887] LNet: 83970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1786.212505] LNetError: 83970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.216718] LNetError: 83970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1786.229786] LNet: Removed LNI 192.168.202.55@tcp [ 1786.246092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.273805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.290729] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1787.204260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1787.371359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1788.792382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.868576] LNet: Removed LNI 192.168.202.55@tcp [ 1789.937807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.085671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1790.122874] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1791.138517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1791.375990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.426965] LNet: Removed LNI 192.168.202.55@tcp [ 1792.499841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.592270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.631928] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1793.251111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1793.357645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1793.615912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.673651] LNet: Removed LNI 192.168.202.55@tcp [ 1794.690802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.731926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1794.740868] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1795.472951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1795.748546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.779396] LNet: 84565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.788448] LNet: 84565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1795.793639] LNetError: 84565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.803616] LNetError: 84565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1795.814369] LNet: Removed LNI 192.168.202.55@tcp [ 1795.833316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.878842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.902934] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1796.665733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1796.763681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1797.025389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.117935] LNet: Removed LNI 192.168.202.55@tcp [ 1798.144379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.211938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1798.266936] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1799.150297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 1799.494210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.553535] LNet: Removed LNI 192.168.202.55@tcp [ 1800.586672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.672917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1800.758907] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1801.676637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1801.745601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.788918] LNet: Removed LNI 192.168.202.55@tcp [ 1801.828435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.882355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.905090] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1802.644674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1802.723354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.767558] LNet: Removed LNI 192.168.202.55@tcp [ 1802.784812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.824903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1802.910567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1802.964868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.018688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1803.092377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.147600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.202233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1803.289815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.337539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.434073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1803.459028] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1804.249026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.289738] LNet: Removed LNI 192.168.202.55@tcp [ 1804.315452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.428088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1814.044188] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:59:25 (1787561965) [ 1814.137641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.154546] LNet: 85872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.158320] LNet: 85872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1814.215076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.346325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1814.387260] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1816.015944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.036996] LNetError: 86017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.043205] LNetError: 86017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1816.056225] LNet: Removed LNI 192.168.202.55@tcp231 [ 1816.085874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.159726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1816.197163] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1816.205311] LNet: 86025:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1816.845290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1816.910158] LNet: Removed LNI 192.168.202.55@tcp231 [ 1816.937337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1816.977403] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1818.949041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.003930] LNet: Removed LNI 192.168.202.55@tcp231 [ 1819.053640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.249311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1819.298925] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/0] [ 1819.321101] LNet: 86178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.950717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1819.978843] LNet: Removed LNI 192.168.202.55@tcp231 [ 1819.991823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1820.016502] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/0] [ 1821.765773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.809595] LNet: Removed LNI 192.168.202.55@tcp231 [ 1821.833668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.972870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1822.029981] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/60] [ 1822.050714] LNet: 86329:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1823.032834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1823.083453] LNet: Removed LNI 192.168.202.55@tcp231 [ 1823.134419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1823.199770] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/60] [ 1825.259543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.313505] LNet: Removed LNI 192.168.202.55@tcp231 [ 1825.382884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.472512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1825.501652] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1825.506576] LNet: 86481:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1825.972474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1826.011724] LNet: Removed LNI 192.168.202.55@tcp231 [ 1826.026911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1826.065207] LNet: Added LNI 192.168.202.55@tcp231 [8/256/0/180] [ 1837.048454] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:59:48 (1787561988) [ 1837.355898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.475600] LNet: Removed LNI 192.168.202.55@tcp231 [ 1837.506361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.551996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1837.566459] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1838.663645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1838.731243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1838.797321] LNet: Removed LNI 192.168.202.55@tcp232 [ 1838.826581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1838.872351] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1839.833561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1839.877880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1839.904741] LNet: Removed LNI 192.168.202.55@tcp232 [ 1839.926830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1839.987593] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1841.198822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1841.233831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1841.276857] LNet: Removed LNI 192.168.202.55@tcp232 [ 1841.305913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1841.346117] LNet: Added LNI 192.168.202.55@tcp232 [8/256/0/180] [ 1842.240694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1842.314420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1842.345656] LNet: Removed LNI 192.168.202.55@tcp232 [ 1842.375357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1851.085921] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:00:02 (1787562002) [ 1852.137246] LNet: 88321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.144930] LNet: 88321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1853.039384] Key type .llcrypt unregistered [ 1853.045145] Key type ._llcrypt unregistered [ 1861.049751] Key type ._llcrypt registered [ 1861.051949] Key type .llcrypt registered [ 1861.207281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1861.226917] alg: No test for adler32 (adler32-zlib) [ 1862.334662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1862.380479] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1872.641017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 1882.777710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.809856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1882.830558] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1885.174617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_interrupt -r 2 [ 1885.209858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 2 [ 1885.256458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_interrupt -r 2 [ 1885.292117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 2 [ 1885.353531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_interrupt -r 2 [ 1885.397144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_interrupt -r 2 [ 1885.435810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_interrupt -r 2 [ 1885.470559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_interrupt -r 2 [ 1885.504726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1885.527891] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1885.534844] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1885.749280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1885.778816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1885.802962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1885.833785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1885.859681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1885.882542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1885.912946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1885.945492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1885.959959] LNet: 88988:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1885.974429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1886.000194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1886.018894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1886.043527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1886.071303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1886.103616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1886.585972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_dropped -r 2 [ 1886.632199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 2 [ 1886.680441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_dropped -r 2 [ 1886.717993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 2 [ 1886.769435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_dropped -r 2 [ 1886.844276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_dropped -r 2 [ 1886.918392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_dropped -r 2 [ 1886.990812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_dropped -r 2 [ 1887.041691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1887.090832] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1887.483376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1887.532895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1887.596498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1887.629527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1887.695175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1887.747497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1887.815693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1887.881573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1887.909421] LNet: 89108:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.944397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1888.002036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1888.063963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1888.098248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1888.145728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1888.199738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1888.816753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_aborted -r 2 [ 1888.867803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 2 [ 1888.930651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_aborted -r 2 [ 1888.983450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 2 [ 1889.020176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_aborted -r 2 [ 1889.071877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_aborted -r 2 [ 1889.146669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_aborted -r 2 [ 1889.195327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_aborted -r 2 [ 1889.256819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1889.296242] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1889.770771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1889.824764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1889.893440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1889.954911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1890.017994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1890.071623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1890.163752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.235212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.266591] LNet: 89229:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.302296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.370773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.436776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.517576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.603568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.685398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1891.804173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_no_route -r 2 [ 1891.873693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 2 [ 1891.940534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_no_route -r 2 [ 1892.010179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 2 [ 1892.117844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_no_route -r 2 [ 1892.226928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_no_route -r 2 [ 1892.299608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_no_route -r 2 [ 1892.389811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_no_route -r 2 [ 1892.464807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1892.506881] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1893.074672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1893.167584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1893.243698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1893.341452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1893.406377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1893.448855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1893.507790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1893.558228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1893.583162] LNet: 89349:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1893.614910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1893.658958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1893.736722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1893.792510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1893.873619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1893.944886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1895.068775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e local_timeout -r 2 [ 1895.164162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 2 [ 1895.239656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e local_timeout -r 2 [ 1895.273326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 2 [ 1895.342553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e local_timeout -r 2 [ 1895.375728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e local_timeout -r 2 [ 1895.407336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e local_timeout -r 2 [ 1895.445788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e local_timeout -r 2 [ 1895.508774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1895.897838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1895.917509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1895.935649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1895.953836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1895.971417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1895.990909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1896.011845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1896.032305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1896.073462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1896.110960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1896.143706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1896.174455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1896.211327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1896.255184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1897.142749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp -m GET -e remote_dropped -r 2 [ 1897.217450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 2 [ 1897.349661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp -m GET -e remote_dropped -r 2 [ 1897.405524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 2 [ 1897.493156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp -m GET -e remote_dropped -r 2 [ 1897.592689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.55@tcp1 -d 192.168.202.155@tcp1 -m GET -e remote_dropped -r 2 [ 1897.652178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp -m GET -e remote_dropped -r 2 [ 1897.694425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.55@tcp1 -d 192.168.202.55@tcp1 -m GET -e remote_dropped -r 2 [ 1897.748185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.55@tcp 192.168.202.155@tcp [ 1897.776924] LNet: There was an unexpected network error while writing to 192.168.202.155: rc = -22 [ 1897.780850] LNet: Skipped 1 previous similar message [ 1897.789397] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 1898.223604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1898.292796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1898.402675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1898.464586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1898.527806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1898.589100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1898.650768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1898.726774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1898.768770] LNet: 89589:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1898.785674] LNet: 89589:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1898.824358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1898.884816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1898.940636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1898.980810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1899.029771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1899.107200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1905.432837] LNet: 89668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.437927] LNetError: 89668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.445127] LNet: Removed LNI 192.168.202.55@tcp [ 1905.464390] LNet: Removed LNI 192.168.202.55@tcp1 [ 1905.912801] Key type .llcrypt unregistered [ 1905.916676] Key type ._llcrypt unregistered [ 1927.368576] Key type ._llcrypt registered [ 1927.372053] Key type .llcrypt registered [ 1928.807811] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:01:20 (1787562080) [ 1929.377430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1929.426900] alg: No test for adler32 (adler32-zlib) [ 1930.487191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.605232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.648328] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1931.994715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1932.040215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.55@tcp [ 1932.079746] LNetError: 90661:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1932.218715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.55@tcp [ 1932.402676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1940.884481] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:01:32 (1787562092) [ 1941.033911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.083267] LNet: 91246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.093919] LNetError: 91246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.111486] LNet: Removed LNI 192.168.202.55@tcp [ 1941.154639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.244435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.263370] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1942.411581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1942.449377] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 1944.013911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.55@tcp [ 1944.045963] LNetError: 91517:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.55@tcp to recovery [ 1944.289280] LNet: 1 local NIs in recovery (showing 1): 192.168.202.55@tcp [ 1944.302665] LNet: There was an unexpected network error while writing to 192.168.202.55: rc = -22 [ 1945.470219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.55@tcp [ 1954.396923] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:01:46 (1787562106) [ 1956.323737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg255-server [ 1957.854066] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:01:50 (1787562110) [ 1959.766160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg255-server [ 1962.056357] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:01:53 (1787562113) [ 1964.886874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg255-server [ 1966.833925] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:01:58 (1787562118) [ 1967.152906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.194843] LNet: 93139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.204708] LNetError: 93139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.216166] LNet: Removed LNI 192.168.202.55@tcp [ 1967.234325] LNet: Removed LNI 192.168.202.55@tcp1 [ 1967.293024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.335516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1967.356326] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1968.364667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1968.483774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1968.554918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1968.643684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1968.696819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1968.805306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1974.851732] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:02:07 (1787562127) [ 1975.219669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.237972] LNet: 93870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.243743] LNetError: 93870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.250795] LNetError: 93870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1975.267994] LNet: Removed LNI 192.168.202.55@tcp [ 1975.289696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.352963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.377460] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1981.906322] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:02:14 (1787562134) [ 1982.355694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.376623] LNet: 94573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.383779] LNetError: 94573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.412237] LNet: Removed LNI 192.168.202.55@tcp [ 1982.456653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.507912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.537156] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 1989.601554] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:02:21 (1787562141) [ 1991.220898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1993.210993] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:02:24 (1787562144) [ 1993.706620] LNet: 95609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.711575] LNetError: 95609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.746290] LNet: Removed LNI 192.168.202.55@tcp [ 1994.504910] Key type .llcrypt unregistered [ 1994.507832] Key type ._llcrypt unregistered [ 2001.247765] Key type ._llcrypt registered [ 2001.251115] Key type .llcrypt registered [ 2001.610900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2001.636668] alg: No test for adler32 (adler32-zlib) [ 2002.592527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2002.625147] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2009.911744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2014.835088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.155@tcp [ 2015.420556] LNet: 96034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.429225] LNetError: 96034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.447952] LNet: Removed LNI 192.168.202.55@tcp [ 2015.961630] Key type .llcrypt unregistered [ 2015.965087] Key type ._llcrypt unregistered [ 2027.356930] Key type ._llcrypt registered [ 2027.359408] Key type .llcrypt registered [ 2028.657235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2030.198501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2032.199302] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:03:04 (1787562184) [ 2033.720410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2035.391501] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:03:07 (1787562187) [ 2037.692487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg255-server [ 2039.372661] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:03:11 (1787562191) [ 2041.618708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg255-server [ 2043.802505] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:03:15 (1787562195) [ 2045.385931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.399839] alg: No test for adler32 (adler32-zlib) [ 2046.623805] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2046.803874] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2048.479250] Key type lgssc registered [ 2049.498484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.918895] Key type lgssc unregistered [ 2072.144054] LNet: 99213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.151941] LNetError: 99213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.184843] LNet: Removed LNI 192.168.202.55@tcp [ 2073.106434] Key type .llcrypt unregistered [ 2073.109391] Key type ._llcrypt unregistered [ 2088.966171] Key type ._llcrypt registered [ 2088.967419] Key type .llcrypt registered [ 2090.220860] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:04:02 (1787562242) [ 2090.810899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2090.835080] alg: No test for adler32 (adler32-zlib) [ 2091.820210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.886211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2091.927383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2091.999716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2092.071198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2099.009530] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:04:10 (1787562250) [ 2099.127352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.169337] LNet: 100624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.207262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.318814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2099.363261] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2100.708057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.735726] LNet: 100774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.749101] LNetError: 100774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.799585] LNet: Removed LNI 192.168.202.55@tcp [ 2100.868281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.927927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2100.964278] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2101.443183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.483523] LNetError: 100802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.502605] LNet: Removed LNI 192.168.202.55@tcp [ 2101.541710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.625595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2101.649827] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2109.297380] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:04:20 (1787562260) [ 2109.486222] LNet: 101386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.506726] LNet: 101386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2109.518281] LNetError: 101386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.573059] LNet: Removed LNI 192.168.202.55@tcp [ 2110.233654] Key type .llcrypt unregistered [ 2110.235216] Key type ._llcrypt unregistered [ 2119.908282] Key type ._llcrypt registered [ 2119.910371] Key type .llcrypt registered [ 2120.081270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.093821] alg: No test for adler32 (adler32-zlib) [ 2121.094890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.174222] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2121.176264] LNet: *** cfs_fail_loc=e003, val=100*** [ 2122.335714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2122.381617] LNet: *** cfs_fail_loc=e003, val=100*** [ 2122.386129] LNet: Skipped 2 previous similar messages [ 2122.400151] Ping reply received with mapping: [ 2122.401814] found metadata reply NI[192.168.202.55@tcp] [ 2122.408631] 37 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2122.414711] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2122.423040] 5a 5a 5a 5a 5a 5a 5a 5a [ 2122.741132] LNet: 101795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.751728] LNetError: 101795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.766836] LNet: Removed LNI 192.168.202.55@tcp [ 2124.064441] Key type .llcrypt unregistered [ 2124.067905] Key type ._llcrypt unregistered [ 2140.886555] Key type ._llcrypt registered [ 2140.896618] Key type .llcrypt registered [ 2143.444165] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:04:53 (1787562293) [ 2144.487508] Key type .llcrypt unregistered [ 2144.492186] Key type ._llcrypt unregistered [ 2153.990090] Key type ._llcrypt registered [ 2153.997710] Key type .llcrypt registered [ 2154.350521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2154.379124] alg: No test for adler32 (adler32-zlib) [ 2155.803695] LNetError: 102857:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2156.330579] Key type .llcrypt unregistered [ 2156.332320] Key type ._llcrypt unregistered [ 2163.085052] Key type ._llcrypt registered [ 2163.097736] Key type .llcrypt registered [ 2164.985264] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:05:16 (1787562316) [ 2165.823981] Key type .llcrypt unregistered [ 2165.825044] Key type ._llcrypt unregistered [ 2173.709663] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2173.712137] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2176.511827] Key type ._llcrypt registered [ 2176.514862] Key type .llcrypt registered [ 2176.828508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2176.867579] alg: No test for adler32 (adler32-zlib) [ 2177.920427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.022540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2189.828935] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:05:41 (1787562341) [ 2190.224270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.248477] LNet: 104647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.256805] LNetError: 104647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2190.275300] LNet: Removed LNI 10.1.2.3@tcp [ 2190.299795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.245353] Key type .llcrypt unregistered [ 2191.248860] Key type ._llcrypt unregistered [ 2199.089097] Key type ._llcrypt registered [ 2199.091748] Key type .llcrypt registered [ 2199.346997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2199.356640] alg: No test for adler32 (adler32-zlib) [ 2200.347299] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2200.455612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2200.603813] LNet: 104940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.612624] LNetError: 104940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.639063] LNet: Removed LNI 192.168.202.55@tcp [ 2202.566579] Key type .llcrypt unregistered [ 2202.568188] Key type ._llcrypt unregistered [ 2216.428455] Key type ._llcrypt registered [ 2216.431616] Key type .llcrypt registered [ 2218.037516] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:06:09 (1787562369) [ 2218.985341] Key type .llcrypt unregistered [ 2218.992137] Key type ._llcrypt unregistered [ 2226.744369] Key type ._llcrypt registered [ 2226.746841] Key type .llcrypt registered [ 2227.110797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2227.139409] alg: No test for adler32 (adler32-zlib) [ 2237.422136] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:06:28 (1787562388) [ 2237.510070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.568047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.688705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.790355] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2239.199309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2239.254625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2239.343408] LNet: Removed LNI 192.168.202.55@tcp [ 2239.829727] LNet: 106700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2240.629256] Key type .llcrypt unregistered [ 2240.633825] Key type ._llcrypt unregistered [ 2257.111301] Key type ._llcrypt registered [ 2257.112403] Key type .llcrypt registered [ 2258.877771] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:06:50 (1787562410) [ 2259.321334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.339712] alg: No test for adler32 (adler32-zlib) [ 2260.402580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.490151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.527370] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2261.917137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2262.044588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.55@tcp [ 2262.077310] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2262.088178] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2262.093371] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2262, 0, 0 [ 2262.100587] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2262.104422] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2262.114554] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2262.118718] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2262, 0, 0 [ 2262.128180] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2262.137383] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2262.146290] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2262.154774] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2262, 0, 2312 [ 2262.163791] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2262.166638] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2262.173763] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2262.178651] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2262, 0, 2312 [ 2262.182840] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2272.463360] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:07:03 (1787562423) [ 2273.027475] LNet: 108241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.040153] LNetError: 108241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.052633] LNet: Removed LNI 192.168.202.55@tcp [ 2274.028785] Key type .llcrypt unregistered [ 2274.035939] Key type ._llcrypt unregistered [ 2282.196787] Key type ._llcrypt registered [ 2282.198627] Key type .llcrypt registered [ 2282.388444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.403628] alg: No test for adler32 (adler32-zlib) [ 2283.447282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2283.497444] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2291.691892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2301.370273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.432237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2301.461206] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 2302.413203] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2302.607972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2302.616337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2302.632457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2305.159880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2305.188467] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2317.054707] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:07:48 (1787562468) [ 2317.365045] LNet: 109754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.373186] LNetError: 109754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.391450] LNet: Removed LNI 192.168.202.55@tcp [ 2318.431482] LNet: Removed LNI 192.168.202.55@tcp1 [ 2318.435185] LNetError: 109754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2318.437614] LNetError: 109754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2318.455182] LNet: Removed LNI 10.1.2.3@tcp99 [ 2319.322615] Key type .llcrypt unregistered [ 2319.324328] Key type ._llcrypt unregistered [ 2327.684471] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2327.689778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2330.458678] Key type ._llcrypt registered [ 2330.460676] Key type .llcrypt registered [ 2330.788344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2330.806961] alg: No test for adler32 (adler32-zlib) [ 2331.753718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.821816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.846899] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2332.876988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2332.935219] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2335.767510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2335.825780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2335.887552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2335.924429] LNetError: 110504:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2335.964965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2336.008995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2336.553966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2336.641229] LNet: 110524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2336.644636] LNetError: 110524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2336.647021] LNet: Removed LNI 10.1.2.3@tcp [ 2337.718280] LNet: Removed LNI 192.168.202.55@tcp [ 2338.722773] Key type .llcrypt unregistered [ 2338.724723] Key type ._llcrypt unregistered [ 2352.319958] Key type ._llcrypt registered [ 2352.325125] Key type .llcrypt registered [ 2354.259567] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:08:25 (1787562505) [ 2354.667040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2354.682673] alg: No test for adler32 (adler32-zlib) [ 2355.582347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.638255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.662886] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2356.760967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2356.956499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg255-client@tcp [ 2367.523415] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:08:38 (1787562518) [ 2367.962065] LNet: 112064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.967697] LNetError: 112064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.981533] LNet: Removed LNI 192.168.202.55@tcp [ 2369.107613] Key type .llcrypt unregistered [ 2369.110551] Key type ._llcrypt unregistered [ 2377.769338] Key type ._llcrypt registered [ 2377.772543] Key type .llcrypt registered [ 2378.090400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2378.120229] alg: No test for adler32 (adler32-zlib) [ 2379.123613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2379.165484] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2379.191820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2386.436297] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:08:58 (1787562538) [ 2389.089172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2391.348228] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:09:02 (1787562542) [ 2393.685212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2395.906925] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:09:07 (1787562547) [ 2396.026626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.062378] LNet: 113483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.077871] LNetError: 113483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.106837] LNet: Removed LNI 192.168.202.55@tcp [ 2396.148814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.252364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.280459] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2397.330312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2397.430334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2404.879550] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:09:16 (1787562556) [ 2405.272420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2405.299706] LNet: 114200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.305159] LNetError: 114200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.331260] LNet: Removed LNI 192.168.202.55@tcp [ 2405.364593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2405.462600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2405.519455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2406.403387] Key type .llcrypt unregistered [ 2406.405620] Key type ._llcrypt unregistered [ 2420.025342] Key type ._llcrypt registered [ 2420.029615] Key type .llcrypt registered [ 2421.875797] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:09:33 (1787562573) [ 2422.481206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2422.507854] alg: No test for adler32 (adler32-zlib) [ 2423.609280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.763965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2423.829105] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2425.048919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2425.139863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2433.934724] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:09:45 (1787562585) [ 2434.075532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2434.111223] LNet: 115752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.122398] LNetError: 115752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.148966] LNet: Removed LNI 192.168.202.55@tcp [ 2434.202926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2434.377986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2434.424723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2444.024940] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:09:55 (1787562595) [ 2444.129826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2444.169450] LNet: 116334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.224790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.384491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2445.054784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2445.123412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2445.315295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2446.115874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2446.159598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2446.283248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.347577] LNet: 116375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.396030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.601373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2447.391783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2447.446548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2447.517687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2448.112315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2448.188896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2456.094271] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:10:07 (1787562607) [ 2456.248647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.264607] LNet: 116976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.280704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2456.332451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2456.367538] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2457.047828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2457.061763] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2457.973330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2458.050332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2458.264476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.287884] LNetError: 117255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.295190] LNet: Removed LNI 192.168.202.55@tcp [ 2459.296995] LNetError: 117255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.342035] LNet: Removed LNI 192.168.202.55@tcp2 [ 2459.409272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2459.491712] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2459.499086] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2459.504803] LNet: 117261:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2460.142560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2460.410511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.427886] LNet: 117298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.431761] LNet: 117298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2461.471419] LNet: Removed LNI 192.168.202.55@tcp [ 2461.475942] LNetError: 117298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.486369] LNetError: 117298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2461.507971] LNet: Removed LNI 192.168.202.55@tcp2 [ 2461.540891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.628400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2461.654087] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2462.639778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2462.678319] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2463.718581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2463.750526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2463.786811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2463.978495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.055215] LNet: Removed LNI 192.168.202.55@tcp [ 2466.086752] LNet: Removed LNI 192.168.202.55@tcp2 [ 2466.119821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.307637] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2466.333284] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2466.342442] LNet: 117589:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2467.081435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2467.357415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.404155] LNetError: 117626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.409613] LNetError: 117626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2467.415661] LNet: Removed LNI 192.168.202.55@tcp [ 2468.454797] LNet: Removed LNI 192.168.202.55@tcp2 [ 2468.513332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.664573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.712966] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2469.681300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2469.697649] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2470.772590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2470.850180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2471.136873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.181380] LNet: 117906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.189703] LNet: 117906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2472.223603] LNet: Removed LNI 192.168.202.55@tcp [ 2472.270902] LNet: Removed LNI 192.168.202.55@tcp2 [ 2472.340580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.427950] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2472.435090] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2472.445686] LNet: 117912:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2473.587616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2473.846530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2474.911989] LNet: Removed LNI 192.168.202.55@tcp [ 2474.939638] LNet: Removed LNI 192.168.202.55@tcp2 [ 2474.969534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.035522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2475.060604] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2475.796244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2475.815352] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2476.540982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2476.573687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2476.643773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2476.872501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.886917] LNetError: 118234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.890775] LNetError: 118234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2476.896562] LNet: Removed LNI 192.168.202.55@tcp [ 2477.932605] LNet: Removed LNI 192.168.202.55@tcp2 [ 2478.024175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.149282] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2478.162842] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2478.169907] LNet: 118240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.918075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2479.130525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2479.154819] LNet: Removed LNI 192.168.202.55@tcp [ 2480.230253] LNet: Removed LNI 192.168.202.55@tcp2 [ 2480.300855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2480.357813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2480.377778] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2481.332822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2481.354219] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2482.490117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2482.534869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2482.608821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2483.040763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.063771] LNet: Removed LNI 192.168.202.55@tcp [ 2484.071827] LNet: Removed LNI 192.168.202.55@tcp2 [ 2484.106144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.217130] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2484.230163] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2484.248201] LNet: 118580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.212837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2485.659731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2486.752046] LNet: Removed LNI 192.168.202.55@tcp [ 2486.765533] LNet: Removed LNI 192.168.202.55@tcp2 [ 2486.796923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.863159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.914531] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2488.107423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2488.139598] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2489.645485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2489.732060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2489.832164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2489.952633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2490.494607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2490.505922] LNet: 118934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.513992] LNet: 118934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2491.551158] LNet: Removed LNI 192.168.202.55@tcp [ 2492.591196] LNet: Removed LNI 192.168.202.55@tcp2 [ 2492.648628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.802206] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2492.807708] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2492.812519] LNet: 118940:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2493.766209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2502.453806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2504.584544] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:10:56 (1787562656) [ 2504.666420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2504.692682] LNetError: 119692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.711100] LNetError: 119692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2504.723162] LNet: Removed LNI 192.168.202.55@tcp [ 2505.778367] LNet: Removed LNI 192.168.202.55@tcp2 [ 2505.826677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.361866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2506.384148] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2507.541455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2507.565923] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2508.857318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2508.907747] LNet: Removed LNI 10.1.2.3@tcp [ 2508.946384] LNet: Removed LNI 192.168.202.55@tcp [ 2509.002657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2509.139443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2509.207370] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2509.212040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2509.541647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2509.578336] LNet: Removed LNI 192.168.202.55@tcp [ 2509.592085] LNet: Removed LNI 10.1.2.3@tcp [ 2518.576614] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 05:11:10 (1787562670) [ 2519.113454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2519.239866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2519.387395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2519.449221] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2521.096961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2521.165180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.253123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.331770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.380137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.481534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.581571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.642288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.765871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2521.896451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2522.006357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.55@tcp [ 2522.400113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2533.654241] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:11:24 (1787562684) [ 2533.829438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.924382] LNet: 121309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.940952] LNet: 121309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2535.020456] LNet: Removed LNI 192.168.202.55@tcp [ 2535.124683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.371435] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2535.674451] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2535.695738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2535.711033] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2538.320131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.344357] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2539.008329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2539.044632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2541.524858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2541.557669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2541.604945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.55@tcp -m PUT -e local_timeout -r 1 [ 2541.673551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2541.737426] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2541.741419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2542.056610] LNetError: 121375:0:(lib-move.c:4801:lnet_parse()) 192.168.202.55@tcp, src 192.168.202.55@tcp: Dropping PUT (error -108 looking up sender) [ 2543.072433] LNetError: 121837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2543.084223] LNetError: 121837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2543.087400] LNet: Removed LNI 10.1.2.3@tcp [ 2544.102701] LNet: Removed LNI 192.168.202.55@tcp [ 2544.608742] Key type .llcrypt unregistered [ 2544.613087] Key type ._llcrypt unregistered [ 2558.410892] Key type ._llcrypt registered [ 2558.421840] Key type .llcrypt registered [ 2561.322386] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:11:51 (1787562711) [ 2561.882304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2561.910873] alg: No test for adler32 (adler32-zlib) [ 2562.931277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2563.573072] LNet: 122692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2564.889394] Key type .llcrypt unregistered [ 2564.894480] Key type ._llcrypt unregistered [ 2572.467549] Key type ._llcrypt registered [ 2572.472507] Key type .llcrypt registered [ 2572.743384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2572.763163] alg: No test for adler32 (adler32-zlib) [ 2573.774865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2573.805234] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2582.305717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2582.555758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2583.455610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.155@tcp [ 2583.800956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2583.834557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2583.878911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2583.919927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2583.964544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2584.010345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2584.058282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2584.095360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2584.096360] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 2584.131594] LNet: 123146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2584.162852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2584.201915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2584.242288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2584.289656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2584.339703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2584.374439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2584.997598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2585.261700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2585.288275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2585.316829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2585.345956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2585.372602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2585.398762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2585.427787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2585.459352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2585.477909] LNet: 123263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2585.493725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2585.523849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2585.554750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2585.591431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2585.620509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2585.653871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2586.348604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.155@tcp [ 2586.591881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2586.625231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2586.655704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2586.687496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2586.710282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2586.739471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2586.766246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2586.794355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2586.810213] LNet: 123380:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2586.826433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2586.853793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2586.886932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2586.915356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2586.943448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2586.972760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2587.638970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2587.896441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2587.975619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2588.031223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2588.094733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2588.137614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2588.180554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2588.275042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2588.416258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2588.491882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2588.560607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2588.629536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2588.719646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2588.786800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2588.833500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2589.224678] LNet: 123547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2589.231841] LNetError: 123547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2589.247989] LNet: Removed LNI 192.168.202.55@tcp [ 2589.665947] Key type .llcrypt unregistered [ 2589.674665] Key type ._llcrypt unregistered [ 2615.726729] Key type ._llcrypt registered [ 2615.730557] Key type .llcrypt registered [ 2617.791501] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:12:48 (1787562768) [ 2618.453570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2618.472500] alg: No test for adler32 (adler32-zlib) [ 2619.390307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2620.005718] LNet: 124436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.950441] Key type .llcrypt unregistered [ 2620.952841] Key type ._llcrypt unregistered [ 2628.195615] Key type ._llcrypt registered [ 2628.197186] Key type .llcrypt registered [ 2628.304126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2628.312249] alg: No test for adler32 (adler32-zlib) [ 2629.194817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2629.217523] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2637.024195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.155@tcp [ 2646.792754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2646.839791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2646.877214] LNet: Added LNI 192.168.202.55@tcp1 [8/256/0/180] [ 2648.390680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2649.357765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.155@tcp [ 2649.710505] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.155@tcp [ 2649.775735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2649.821648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2649.855900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2649.914792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2649.957224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2650.019606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2650.076357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2650.132558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2650.159217] LNet: 125091:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2650.191692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2650.234777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2650.273878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2650.314936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2650.348896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2650.385656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2650.974968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2651.210174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2651.230788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2651.333671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2651.372669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2651.402171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2651.453805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2651.524916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2651.558478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2651.573173] LNet: 125208:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2651.593382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2651.643131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2651.671564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2651.695593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2651.729288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2651.767464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2652.792072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.155@tcp [ 2653.167186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2653.190648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2653.213841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2653.247965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2653.303925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2653.349145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2653.409922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2653.435367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2653.452769] LNet: 125326:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2653.479618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2653.533717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2653.604709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2653.651941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2653.682310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2653.705648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2654.273546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2654.512724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2654.534956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2654.560909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2654.587374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2654.607323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2654.629722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2654.646203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2654.662336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2654.680914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2654.702961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2654.727871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2654.753577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2654.785552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2654.828381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2660.680978] LNet: 125528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.698725] LNetError: 125528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.713958] LNet: Removed LNI 192.168.202.55@tcp [ 2660.729622] LNet: Removed LNI 192.168.202.55@tcp1 [ 2661.591145] Key type .llcrypt unregistered [ 2661.599265] Key type ._llcrypt unregistered [ 2685.559804] Key type ._llcrypt registered [ 2685.562675] Key type .llcrypt registered [ 2687.749571] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 05:13:58 (1787562838) [ 2688.344492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2688.358421] alg: No test for adler32 (adler32-zlib) [ 2689.401460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2689.465962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2689.509487] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2690.683824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2690.702805] LNet: Added LNI 192.168.202.55@tcp2 [8/256/0/180] [ 2691.558298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.55@tcp [ 2691.660988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2692.706272] LNet: Removed LNI 192.168.202.55@tcp [ 2699.759638] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 05:14:11 (1787562851) [ 2701.101843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2703.200988] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:14:14 (1787562854) [ 2713.545213] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:14:23 (1787562863) [ 2724.310371] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:14:35 (1787562875) [ 2736.526595] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:14:48 (1787562888) [ 2748.073933] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:14:59 (1787562899) [ 2755.821808] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:15:08 (1787562908) [ 2765.516470] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:15:17 (1787562917) [ 2773.869091] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:15:26 (1787562926) [ 2783.468460] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:15:35 (1787562935) [ 2793.380452] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:15:44 (1787562944) [ 2800.934951] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:15:53 (1787562953) [ 2810.748526] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:16:01 (1787562961) [ 2819.573172] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:16:11 (1787562971) [ 2829.068599] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:16:20 (1787562980) [ 2836.633896] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:16:28 (1787562988) [ 2847.586467] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:16:39 (1787562999) [ 2854.628378] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:16:46 (1787563006) [ 2862.135207] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:16:54 (1787563014) [ 2862.554953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2862.612262] LNet: 137228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2862.635861] LNetError: 137228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2862.671352] LNet: Removed LNI 192.168.202.55@tcp2 [ 2862.733599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2862.977508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2863.016748] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2864.679254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2864.693509] LNet: 137370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2864.698174] LNetError: 137370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2864.712888] LNet: Removed LNI 192.168.202.55@tcp [ 2864.727632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2864.767590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2864.788910] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 2872.904933] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:17:04 (1787563024) [ 2880.525731] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:17:12 (1787563032) [ 2888.412716] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:17:20 (1787563040) [ 2898.394516] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:17:29 (1787563049) [ 2909.667498] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:17:40 (1787563060) [ 2922.232512] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:17:54 (1787563074) [ 2934.667161] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:18:06 (1787563086) [ 2942.644383] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:18:14 (1787563094) [ 2951.028717] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:18:23 (1787563103) [ 2960.535926] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:18:31 (1787563111) [ 2971.824721] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:18:42 (1787563122) [ 2985.019599] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:18:55 (1787563135) [ 2995.812749] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2735 sec ========== 05:19:07 (1787563147) [ 2996.350682] LNet: 144841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.357515] LNetError: 144841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2996.385019] LNet: Removed LNI 192.168.202.55@tcp [ 2997.373757] Key type .llcrypt unregistered [ 2997.378852] Key type ._llcrypt unregistered [ 3013.977799] Key type ._llcrypt registered [ 3013.981089] Key type .llcrypt registered [ 3014.230389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3014.254649] alg: No test for adler32 (adler32-zlib) [ 3015.355803] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3015.682286] LNet: Added LNI 192.168.202.55@tcp [8/256/0/180] [ 3017.407773] Key type lgssc registered [ 3019.274555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3099.079680] Lustre: Mounted lustre-client - version 2.17.57_82_g4029ffa [ 3104.686493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3114.687224] Lustre: Unmounted lustre-client [ 3148.748861] Key type lgssc unregistered [ 3149.362312] LNet: 147803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3149.394141] LNetError: 147803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.445052] LNet: Removed LNI 192.168.202.55@tcp [ 3150.548342] Key type .llcrypt unregistered [ 3150.549782] Key type ._llcrypt unregistered