[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470893580 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006900] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008000] LSM: Security Framework initializing [ 0.008050] Yama: becoming mindful. [ 0.010004] SELinux: Initializing. [ 0.011081] *** VALIDATE selinux *** [ 0.019176] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024385] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027144] *** VALIDATE tmpfs *** [ 0.029471] *** VALIDATE proc *** [ 0.030264] *** VALIDATE cgroup *** [ 0.031012] *** VALIDATE cgroup2 *** [ 0.032261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035035] Spectre V2 : User space: Vulnerable [ 0.036022] Speculative Store Bypass: Vulnerable [ 0.038773] debug: unmapping init [mem 0xffffffff91059000-0xffffffff91060fff] [ 0.041999] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042668] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046012] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048019] ... fixed-purpose events: 3 [ 0.049014] ... event mask: 000000070000000f [ 0.050326] rcu: Hierarchical SRCU implementation. [ 0.052494] smp: Bringing up secondary CPUs ... [ 0.053591] x86: Booting SMP configuration: [ 0.054023] .... node #0, CPUs: #1 #2 #3 [ 0.057284] smp: Brought up 1 node, 4 CPUs [ 0.059013] smpboot: Max logical packages: 1 [ 0.060014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.109614] node 0 deferred pages initialised in 47ms [ 0.111118] devtmpfs: initialized [ 0.113236] x86/mm: Memory block size: 128MB [ 0.116021] gcov: version magic: 0x41383552 [ 0.118228] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120274] pinctrl core: initialized pinctrl subsystem [ 0.121163] [ 0.121724] ************************************************************* [ 0.122010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123015] ** ** [ 0.124014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125011] ** ** [ 0.126012] ** This means that this kernel is built to expose internal ** [ 0.127010] ** IOMMU data structures, which may compromise security on ** [ 0.128008] ** your system. ** [ 0.129008] ** ** [ 0.130011] ** If you see this message and you are not debugging the ** [ 0.131009] ** kernel, report this immediately to your vendor! ** [ 0.132011] ** ** [ 0.133008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134010] ************************************************************* [ 0.135729] NET: Registered protocol family 16 [ 0.136454] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140044] cpuidle: using governor menu [ 0.141940] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.145444] PCI: Using configuration type 1 for base access [ 0.147127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.156244] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.158016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.160204] cryptd: max_cpu_qlen set to 1000 [ 0.162381] ACPI: Added _OSI(Module Device) [ 0.163013] ACPI: Added _OSI(Processor Device) [ 0.164007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169495] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.174617] ACPI: Interpreter enabled [ 0.176080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179018] ACPI: Using IOAPIC for interrupt routing [ 0.181133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196752] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.200048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.204030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.209142] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.214126] acpiphp: Slot [2] registered [ 0.215073] acpiphp: Slot [5] registered [ 0.216075] acpiphp: Slot [6] registered [ 0.217045] acpiphp: Slot [3] registered [ 0.218057] acpiphp: Slot [4] registered [ 0.219092] acpiphp: Slot [7] registered [ 0.220134] acpiphp: Slot [8] registered [ 0.222160] acpiphp: Slot [9] registered [ 0.224117] acpiphp: Slot [10] registered [ 0.225108] acpiphp: Slot [11] registered [ 0.227112] acpiphp: Slot [12] registered [ 0.228081] acpiphp: Slot [13] registered [ 0.230118] acpiphp: Slot [14] registered [ 0.232102] acpiphp: Slot [15] registered [ 0.234160] acpiphp: Slot [16] registered [ 0.236133] acpiphp: Slot [17] registered [ 0.238121] acpiphp: Slot [18] registered [ 0.240103] acpiphp: Slot [19] registered [ 0.241115] acpiphp: Slot [20] registered [ 0.243104] acpiphp: Slot [21] registered [ 0.244128] acpiphp: Slot [22] registered [ 0.247117] acpiphp: Slot [23] registered [ 0.248120] acpiphp: Slot [24] registered [ 0.250112] acpiphp: Slot [25] registered [ 0.252140] acpiphp: Slot [26] registered [ 0.254129] acpiphp: Slot [27] registered [ 0.256120] acpiphp: Slot [28] registered [ 0.257114] acpiphp: Slot [29] registered [ 0.260140] acpiphp: Slot [30] registered [ 0.261106] acpiphp: Slot [31] registered [ 0.263085] PCI host bridge to bus 0000:00 [ 0.264023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.266027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.271025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283073] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.284947] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290414] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.293439] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.295012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.296010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.298020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.300501] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.302806] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.306049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.309882] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.313834] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.322885] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.327015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332637] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.342016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.349021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.365025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.373753] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.383018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.392021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.406028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.418011] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419557] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.421419] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.424517] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.426256] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.432030] iommu: Default domain type: Passthrough [ 0.434650] SCSI subsystem initialized [ 0.436226] ACPI: bus type USB registered [ 0.437136] usbcore: registered new interface driver usbfs [ 0.440133] usbcore: registered new interface driver hub [ 0.442156] usbcore: registered new device driver usb [ 0.444235] pps_core: LinuxPPS API ver. 1 registered [ 0.446020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.449096] PTP clock support registered [ 0.452154] EDAC MC: Ver: 3.0.0 [ 0.453595] PCI: Using ACPI for IRQ routing [ 0.456148] NetLabel: Initializing [ 0.458016] NetLabel: domain hash size = 128 [ 0.460014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.463099] NetLabel: unlabeled traffic allowed by default [ 0.465161] vgaarb: loaded [ 0.466433] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.469027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.475226] clocksource: Switched to clocksource kvm-clock [ 0.585685] VFS: Disk quotas dquot_6.6.0 [ 0.587255] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.589784] *** VALIDATE ramfs *** [ 0.590949] *** VALIDATE hugetlbfs *** [ 0.593299] pnp: PnP ACPI init [ 0.595781] pnp: PnP ACPI: found 6 devices [ 0.611429] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.614681] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.616676] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.618771] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.621131] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.623382] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.626209] NET: Registered protocol family 2 [ 0.628606] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.633724] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.637297] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.642553] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.645770] TCP: Hash tables configured (established 65536 bind 65536) [ 0.648597] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.651594] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.654345] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657404] NET: Registered protocol family 1 [ 0.661368] RPC: Registered named UNIX socket transport module. [ 0.663436] RPC: Registered udp transport module. [ 0.665381] RPC: Registered tcp transport module. [ 0.666995] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.669301] NET: Registered protocol family 44 [ 0.670778] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.672902] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.674962] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.677138] PCI: CLS 0 bytes, default 64 [ 0.679086] Unpacking initramfs... [ 2.101830] debug: unmapping init [mem 0xffff9b9c3cc64000-0xffff9b9c3ffcffff] [ 2.107583] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.109560] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.111923] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.561863] Initialise system trusted keyrings [ 2.563612] Key type blacklist registered [ 2.565448] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.573619] zbud: loaded [ 2.576077] *** VALIDATE nfs *** [ 2.577164] *** VALIDATE nfs4 *** [ 2.578643] pstore: using deflate compression [ 2.581952] Platform Keyring initialized [ 2.680549] NET: Registered protocol family 38 [ 2.682274] Key type asymmetric registered [ 2.683673] Asymmetric key parser 'x509' registered [ 2.685506] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.689088] io scheduler mq-deadline registered [ 2.691406] io scheduler kyber registered [ 2.693572] io scheduler bfq registered [ 2.697036] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.700625] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.702916] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.705662] ACPI: Power Button [PWRF] [ 2.710627] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.715734] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.724823] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.749680] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.776578] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.780854] Non-volatile memory driver v1.3 [ 2.782395] Linux agpgart interface v0.103 [ 2.809272] virtio_blk virtio1: [vda] 149960 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.811753] vda: detected capacity change from 0 to 76779520 [ 2.826504] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.829745] vdb: detected capacity change from 0 to 1073741824 [ 2.840412] libphy: Fixed MDIO Bus: probed [ 2.848286] usbcore: registered new interface driver usbserial_generic [ 2.853333] usbserial: USB Serial support registered for generic [ 2.855397] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.858610] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.860334] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.862936] mousedev: PS/2 mouse device common for all mice [ 2.866718] rtc_cmos 00:05: RTC can wake from S4 [ 2.869695] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.874638] rtc_cmos 00:05: registered as rtc0 [ 2.876530] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.877279] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.884288] intel_pstate: CPU model not supported [ 2.886246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.892043] hid: raw HID events driver (C) Jiri Kosina [ 2.894471] usbcore: registered new interface driver usbhid [ 2.896315] usbhid: USB HID core driver [ 2.898073] drop_monitor: Initializing network drop monitor service [ 2.900519] Initializing XFRM netlink socket [ 2.901843] NET: Registered protocol family 10 [ 2.904486] Segment Routing with IPv6 [ 2.905617] NET: Registered protocol family 17 [ 2.907222] mpls_gso: MPLS GSO support [ 2.912177] RAS: Correctable Errors collector initialized. [ 2.914295] AVX version of gcm_enc/dec engaged. [ 2.915315] AES CTR mode by8 optimization enabled [ 3.014972] sched_clock: Marking stable (3014946189, 0)->(3928772606, -913826417) [ 3.019112] registered taskstats version 1 [ 3.022199] Loading compiled-in X.509 certificates [ 3.024821] zswap: loaded using pool lzo/zbud [ 3.052879] Key type big_key registered [ 3.064524] Key type encrypted registered [ 3.065965] ima: No TPM chip found, activating TPM-bypass! [ 3.068160] ima: Allocated hash algorithm: sha1 [ 3.069462] ima: No architecture policies found [ 3.070946] evm: Initialising EVM extended attributes: [ 3.072601] evm: security.selinux [ 3.073777] evm: security.ima [ 3.074684] evm: security.capability [ 3.075743] evm: HMAC attrs: 0x1 [ 3.078605] rtc_cmos 00:05: setting system clock to 2026-09-16 00:13:22 UTC (1789517602) [ 3.084940] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 3.088957] debug: unmapping init [mem 0xffffffff90d82000-0xffffffff91058fff] [ 3.099104] Write protecting the kernel read-only data: 28672k [ 3.102949] debug: unmapping init [mem 0xffffffff8f403000-0xffffffff8f5fffff] [ 3.106231] debug: unmapping init [mem 0xffffffff8fd14000-0xffffffff8fdfffff] [ 3.140651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.149060] systemd[1]: Detected virtualization kvm. [ 3.150401] systemd[1]: Detected architecture x86-64. [ 3.151693] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.181822] systemd[1]: No hostname configured. [ 3.183713] systemd[1]: Set hostname to . [ 3.185683] random: systemd: uninitialized urandom read (16 bytes read) [ 3.188445] systemd[1]: Initializing machine ID from random generator. [ 3.229256] random: ln: uninitialized urandom read (6 bytes read) [ 3.320301] random: systemd: uninitialized urandom read (16 bytes read) [ 3.323554] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.329202] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.335378] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.873892] device-mapper: uevent: version 1.0.3 [ 3.875661] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.505262] virtio_net virtio0 ens2: renamed from eth0 [ 4.524192] random: fast init done [ 4.612732] scsi host0: ata_piix [ 4.637975] scsi host1: ata_piix [ 4.639430] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.641522] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.119056] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.583874] random: crng init done [ 9.584836] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.766708] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.749345] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.959923] SELinux: Disabled at runtime. [ 11.011931] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.018135] systemd[1]: Detected virtualization kvm. [ 11.019318] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.471811] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.474420] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.478300] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.480732] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.482747] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.491529] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.495198] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ 11.580048] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 11.931534] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.208236] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.208920] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.299397] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.311786] EDAC sbridge: Ver: 1.1.2 [ 13.266705] Key type dns_resolver registered [ 13.575659] NFS: Registering the id_resolver key type [ 13.577508] Key type id_resolver registered [ 13.579323] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-client login: [ 45.289095] hrtimer: interrupt took 4179649 ns [ 71.101968] libcfs: loading out-of-tree module taints kernel. [ 71.164888] Key type ._llcrypt registered [ 71.166508] Key type .llcrypt registered [ 72.059717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 72.071984] alg: No test for adler32 (adler32-zlib) [ 73.656417] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 74.802229] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 76.650276] Key type lgssc registered [ 78.253599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 196.922257] Lustre: Mounted lustre-client - version 2.17.58_39_g97904ba [ 202.365279] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 217.798530] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing check_logdir /tmp/testlogs/ [ 222.688175] Lustre: lustre-OST0000-osc-ffff9b9c88071800: disconnect after 24s idle [ 223.194839] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing yml_node [ 227.737732] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 230.589227] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 232.851624] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 234.807149] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 15 20:17:12 EDT 2026 [ 251.271526] Lustre: Unmounted lustre-client [ 288.299448] Key type lgssc unregistered [ 288.526975] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.532525] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.556165] LNet: Removed LNI 192.168.201.22@tcp [ 289.527153] Key type .llcrypt unregistered [ 289.530707] Key type ._llcrypt unregistered [ 320.232520] Key type ._llcrypt registered [ 320.236335] Key type .llcrypt registered [ 320.407653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.421647] alg: No test for adler32 (adler32-zlib) [ 321.388907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 321.481457] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 321.505836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 325.818864] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 326.023724] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.032077] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.051957] LNet: Removed LNI 192.168.201.22@tcp [ 326.912966] Key type .llcrypt unregistered [ 326.920107] Key type ._llcrypt unregistered [ 334.201769] Key type ._llcrypt registered [ 334.204553] Key type .llcrypt registered [ 335.940456] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:18:53 (1789517933) [ 336.339197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 336.359282] alg: No test for adler32 (adler32-zlib) [ 337.408746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.496381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.512862] LNet: 9709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.536328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.594467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 337.620354] LNet: 9716:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 345.258877] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:19:03 (1789517943) [ 345.346224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.369303] LNet: 10290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.395078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.489868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 352.308503] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:19:10 (1789517950) [ 352.645618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.663928] LNet: 10882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.679658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.765930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 359.093792] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:19:17 (1789517957) [ 359.229437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.249659] LNet: 11466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.281519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.345601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 364.680510] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:19:23 (1789517963) [ 364.983745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.000284] LNet: 12049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.026767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.094112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 365.319960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 371.136194] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:19:29 (1789517969) [ 371.268894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.321144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.416417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 371.831571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 372.204546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 377.859043] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:19:36 (1789517976) [ 378.037029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.062384] LNet: 13236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.076543] LNet: 13236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 378.110399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.199784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 384.841407] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:19:43 (1789517983) [ 385.060271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.090720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.149051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 385.184417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.479559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 385.512711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 385.812246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 385.845202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 386.169388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 386.195944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 386.610394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 386.648560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 387.002879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 387.035578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.336295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 387.367759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 387.684764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 387.704475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.910516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 387.935399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 388.294411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 388.342759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.697282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 388.756564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 389.227821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 389.263505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.585504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.633740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 394.875805] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:19:53 (1789517993) [ 395.056786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.100729] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.109547] LNet: 14502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 395.129391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.195500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 395.235252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 401.179630] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:19:59 (1789517999) [ 401.489925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.538664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.625171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 401.666568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 407.905461] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:20:05 (1789518005) [ 408.014037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.058119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.157096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 408.182859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 414.657755] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:20:12 (1789518012) [ 414.930274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.000893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.062800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 415.084195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.207076] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:20:19 (1789518019) [ 421.325960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.378316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.472352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.499157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 427.632041] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:20:25 (1789518025) [ 427.883677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.902976] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.907711] LNet: 17433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 427.932468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.003855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.033970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 434.078252] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:20:32 (1789518032) [ 434.142284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.197578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.254847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.783166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.371677] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:20:39 (1789518039) [ 441.530581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.591597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.847380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.174541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.854589] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:20:46 (1789518046) [ 448.005269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.059431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.220786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.453550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.524161] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:20:52 (1789518052) [ 454.606535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.677095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.350813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.732105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.159591] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:21:00 (1789518060) [ 462.281592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.336130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.448778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.488470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.817248] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:21:06 (1789518066) [ 468.939314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.959222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.997739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.021424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.612785] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:21:13 (1789518073) [ 475.728202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.763922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.910506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 481.793488] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:21:20 (1789518080) [ 482.006266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.092334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.153137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 487.903841] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:21:26 (1789518086) [ 488.070537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.139641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.253687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 488.706188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 495.647610] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:21:33 (1789518093) [ 495.793895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.837400] LNet: 23455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.841884] LNet: 23455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 495.898575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.965835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 495.991819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 502.773209] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:21:40 (1789518100) [ 502.961701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.045631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.182342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 503.218424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 510.457438] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:21:48 (1789518108) [ 510.555383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.612312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.705797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 510.730668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 516.667487] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:21:54 (1789518114) [ 516.844590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.869469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.924445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.966515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 523.354567] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:22:01 (1789518121) [ 523.594194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.616988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.649785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 523.720990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 523.751684] LNetError: 25818:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 523.798318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 523.835222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 529.451821] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:22:07 (1789518127) [ 529.524538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.584705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.703659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pdSBm3B6DQ [ 529.778499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pdSBm3B6DQ [ 529.823431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pdSBm3B6DQ [ 536.096495] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:22:13 (1789518133) [ 536.171498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.212737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.286358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 536.339404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 536.383438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 541.929484] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:22:20 (1789518140) [ 542.249957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.285659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.367577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.386110] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 543.658541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 544.609384] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 546.527286] Key type lgssc registered [ 548.213774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 552.929820] LNetError: 27579:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.22@tcp99 has no available nets [ 556.043833] Key type lgssc unregistered [ 556.392962] LNetError: 28684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.414595] LNet: Removed LNI 192.168.201.22@tcp [ 558.374443] Key type .llcrypt unregistered [ 558.382214] Key type ._llcrypt unregistered [ 567.418937] Key type ._llcrypt registered [ 567.433497] Key type .llcrypt registered [ 568.717978] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:22:46 (1789518166) [ 569.402471] Key type .llcrypt unregistered [ 569.410659] Key type ._llcrypt unregistered [ 575.528348] Key type ._llcrypt registered [ 575.530574] Key type .llcrypt registered [ 575.748287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 575.764345] alg: No test for adler32 (adler32-zlib) [ 576.664591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.774538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.817864] LNet: 29594:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 576.831705] LNet: 29594:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 576.842768] LNet: 29594:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 576.866253] LNet: Message forwarding will be enabled in 2s [ 579.935259] LNet: Message forwarding enabled [ 580.283792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 580.312629] LNet: Message forwarding will stop in 2s to 6s [ 584.033284] LNet: Message forwarding disabled [ 585.190378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 585.238884] LNet: 29692:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 585.253461] LNet: 29692:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 585.282296] LNet: 29692:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 585.298219] LNet: Message forwarding will be enabled in 2s [ 585.536932] LNet: 29705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.168463] Key type .llcrypt unregistered [ 586.170158] Key type ._llcrypt unregistered [ 600.964733] Key type ._llcrypt registered [ 600.967683] Key type .llcrypt registered [ 602.368621] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:23:20 (1789518200) [ 602.827062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.837482] alg: No test for adler32 (adler32-zlib) [ 603.685818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.768366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 610.074632] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:23:27 (1789518207) [ 610.245105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.258499] LNet: 31105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.287550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.352883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 610.415710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 610.465600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 610.484963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 610.539760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 610.556836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 610.592392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 610.627849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 610.659437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 610.688266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 610.715996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 610.749682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 610.783809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 610.816668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 610.848342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 610.894726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 610.944760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 610.978400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 611.002836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 611.018537] LNetError: 31148:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 617.405958] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:23:35 (1789518215) [ 617.524606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.561237] LNet: 31711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.595904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.712123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.004924] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:23:41 (1789518221) [ 624.668530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg122-server [ 627.064539] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:23:44 (1789518224) [ 627.171766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.219442] LNet: 32642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.251454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.340696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.372310] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 628.135184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[23-27]@tcp [ 628.154196] LNet: 32782:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 628.205117] LNetError: 32652:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 628.226137] LNetError: 32652:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.23@tcp has gone from up to down [ 634.199201] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:23:52 (1789518232) [ 636.316975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg122-server [ 638.165326] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:23:56 (1789518236) [ 638.312719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.342297] LNet: 33712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.353049] LNetError: 33712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.371080] LNet: Removed LNI 192.168.201.22@tcp [ 638.388043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.462903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 638.482119] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 639.078152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 639.107806] LNet: 33835:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 639.114494] LNet: 33835:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 639.137619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 639.159759] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 639.181919] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 639.189171] LNetError: 33722:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.23@tcp has gone from up to down [ 639.196116] LNetError: 33722:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 645.298372] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:24:03 (1789518243) [ 645.499827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.524519] LNet: 34427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.537359] LNetError: 34427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.577872] LNet: Removed LNI 192.168.201.22@tcp [ 645.624225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.698013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 645.746623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.767709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.808452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.843362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.868587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.908790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 645.952160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.981081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.030882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.107305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.163294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.228802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 646.276740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.300870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.347245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.427536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.460718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.509853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 646.575020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.627760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.678914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.752712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.786310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.839343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.915147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.943332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.990923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.045126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.070570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.105879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 647.147518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.176818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.228463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.993914] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:24:11 (1789518251) [ 654.540353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg122-server [ 656.119787] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:24:14 (1789518254) [ 657.751485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg122-server [ 659.314933] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:24:17 (1789518257) [ 659.406115] LNet: 35820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.412702] LNet: 35820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 660.111285] Key type .llcrypt unregistered [ 660.124078] Key type ._llcrypt unregistered [ 666.449133] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.666425] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.672450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.681353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 668.236874] Key type ._llcrypt registered [ 668.239477] Key type .llcrypt registered [ 668.454077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.484505] alg: No test for adler32 (adler32-zlib) [ 669.501964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.574831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.639373] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 670.465753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 670.491861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 672.609244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 672.633819] LNet: Removed LNI 10.1.2.3@tcp [ 672.792686] LNet: 36532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.799117] LNetError: 36532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.815343] LNet: Removed LNI 192.168.201.22@tcp [ 673.289095] Key type .llcrypt unregistered [ 673.292504] Key type ._llcrypt unregistered [ 683.118603] Key type ._llcrypt registered [ 683.120992] Key type .llcrypt registered [ 684.109595] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:24:42 (1789518282) [ 684.595631] Key type .llcrypt unregistered [ 684.599946] Key type ._llcrypt unregistered [ 689.863329] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 689.993863] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.000148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.013696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 692.668356] Key type ._llcrypt registered [ 692.673452] Key type .llcrypt registered [ 692.934853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.948553] alg: No test for adler32 (adler32-zlib) [ 693.860303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.927269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.946621] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 694.572542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 694.593695] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.023818] LNet: 38016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.031381] LNetError: 38016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 697.038807] LNet: Removed LNI 10.1.2.3@tcp [ 697.045731] LNet: Removed LNI 192.168.201.22@tcp [ 697.507470] Key type .llcrypt unregistered [ 697.509099] Key type ._llcrypt unregistered [ 708.878733] Key type ._llcrypt registered [ 708.880770] Key type .llcrypt registered [ 710.530705] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:25:08 (1789518308) [ 710.958376] Key type .llcrypt unregistered [ 710.960356] Key type ._llcrypt unregistered [ 717.488541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 717.737603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.742696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 717.750617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.289578] Key type ._llcrypt registered [ 720.290910] Key type .llcrypt registered [ 720.520287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 720.532102] alg: No test for adler32 (adler32-zlib) [ 721.471309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.526792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 721.566694] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 723.003676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.022302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.334071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.359803] LNet: Removed LNI 10.1.2.3@tcp [ 725.382070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 725.427725] LNet: Removed LNI 10.1.2.31@tcp [ 725.450794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 725.473677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.748385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 726.778406] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 728.067311] LNet: 39930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.083783] LNetError: 39930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 728.091766] LNet: Removed LNI 10.1.2.31@tcp [ 728.112876] LNet: Removed LNI 10.1.2.3@tcp [ 729.222942] Key type .llcrypt unregistered [ 729.227107] Key type ._llcrypt unregistered [ 740.992644] Key type ._llcrypt registered [ 740.997174] Key type .llcrypt registered [ 742.475821] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:25:40 (1789518340) [ 743.194444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 743.211542] alg: No test for adler32 (adler32-zlib) [ 744.203255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.291667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.323033] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 745.181857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.214084] LNet: 40880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.219518] LNetError: 40880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.236394] LNet: Removed LNI 192.168.201.22@tcp [ 745.273338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.387495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.22@tcp [ 745.444733] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 746.418640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.22@tcp [ 746.452047] LNet: Removed LNI 192.168.201.22@tcp [ 746.539648] LNet: 41015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.182666] Key type .llcrypt unregistered [ 747.184868] Key type ._llcrypt unregistered [ 759.069334] Key type ._llcrypt registered [ 759.071652] Key type .llcrypt registered [ 760.171234] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:25:58 (1789518358) [ 761.517693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg122-server [ 763.108724] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:26:01 (1789518361) [ 763.433245] Key type .llcrypt unregistered [ 763.436967] Key type ._llcrypt unregistered [ 768.297189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.426092] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.439881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.449146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.856839] Key type ._llcrypt registered [ 770.859591] Key type .llcrypt registered [ 771.058877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.081889] alg: No test for adler32 (adler32-zlib) [ 771.966408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.045147] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 772.618849] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 774.328617] LNet: 42846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.339761] LNetError: 42846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.344257] LNet: Removed LNI 192.168.201.22@tcp1 [ 774.361776] LNet: Removed LNI 10.1.2.3@tcp2 [ 774.919640] Key type .llcrypt unregistered [ 774.921881] Key type ._llcrypt unregistered [ 784.418754] Key type ._llcrypt registered [ 784.421326] Key type .llcrypt registered [ 785.812630] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:26:23 (1789518383) [ 786.375985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 786.405584] alg: No test for adler32 (adler32-zlib) [ 787.357368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.462659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.511268] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 788.412675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.440989] LNet: 43813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.450798] LNetError: 43813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.463410] LNet: Removed LNI 192.168.201.22@tcp [ 788.486561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.562423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 788.602929] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 796.176497] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:26:33 (1789518393) [ 796.298442] LNet: 44395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.304456] LNetError: 44395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.322992] LNet: Removed LNI 192.168.201.22@tcp [ 797.290518] Key type .llcrypt unregistered [ 797.292975] Key type ._llcrypt unregistered [ 803.038330] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.207652] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.211898] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.218742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.837731] Key type ._llcrypt registered [ 805.840725] Key type .llcrypt registered [ 806.251892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.290028] alg: No test for adler32 (adler32-zlib) [ 807.383313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.487173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.570302] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 808.117363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 808.144764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.656523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.684701] LNet: 45143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.694050] LNetError: 45143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.697253] LNet: Removed LNI 10.1.2.3@tcp [ 810.711391] LNet: Removed LNI 192.168.201.22@tcp [ 810.737925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.813734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 810.847467] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 810.851965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 811.343365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.365743] LNet: 45170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.372287] LNetError: 45170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 811.377574] LNetError: 45170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 811.386582] LNet: Removed LNI 10.1.2.3@tcp [ 811.409781] LNet: Removed LNI 192.168.201.22@tcp [ 811.467627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.561717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 811.581926] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 811.586303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 812.113366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 812.127149] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 813.069953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.094876] LNet: 45332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.110702] LNetError: 45332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 813.115242] LNetError: 45332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 813.119783] LNet: Removed LNI 10.1.2.3@tcp [ 813.124176] LNet: Removed LNI 192.168.201.22@tcp [ 813.134439] LNet: Removed LNI 192.168.201.22@tcp2 [ 813.159772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.218359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 813.239170] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 813.244528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.248691] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 820.131565] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:26:58 (1789518418) [ 820.416223] LNet: 45929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.423377] LNetError: 45929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.434421] LNetError: 45929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 820.440321] LNet: Removed LNI 10.1.2.3@tcp [ 820.443937] LNet: Removed LNI 192.168.201.22@tcp [ 820.454441] LNet: Removed LNI 192.168.201.22@tcp2 [ 821.024789] Key type .llcrypt unregistered [ 821.032596] Key type ._llcrypt unregistered [ 827.365842] Key type ._llcrypt registered [ 827.367763] Key type .llcrypt registered [ 827.625974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.648454] alg: No test for adler32 (adler32-zlib) [ 828.538931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.590411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.610858] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 829.281892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.298366] LNet: 46321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.308452] LNetError: 46321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.320889] LNet: Removed LNI 192.168.201.22@tcp [ 829.334280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.380601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 829.401446] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 829.969455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.021438] LNet: 46351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.029813] LNetError: 46351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.048736] LNet: Removed LNI 192.168.201.22@tcp [ 830.069977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.144704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 830.180934] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 830.804450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.835975] LNet: Removed LNI 192.168.201.22@tcp [ 830.860237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.955383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 830.998928] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 831.522158] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 831.710894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 831.720191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 831.726038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 834.107905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.133329] LNet: 46459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.136800] LNet: 46459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 834.145251] LNetError: 46459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.148597] LNetError: 46459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 834.173977] LNet: Removed LNI 192.168.201.22@tcp [ 834.210511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.293474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 834.330795] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 841.302583] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:27:19 (1789518439) [ 841.661303] LNet: 47056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.668446] LNetError: 47056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.695301] LNet: Removed LNI 192.168.201.22@tcp [ 842.304099] Key type .llcrypt unregistered [ 842.310526] Key type ._llcrypt unregistered [ 849.703657] Key type ._llcrypt registered [ 849.705720] Key type .llcrypt registered [ 849.945954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.969841] alg: No test for adler32 (adler32-zlib) [ 850.930911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.998382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.021802] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 851.871348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.887786] LNet: 47452:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.892978] LNetError: 47452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.902798] LNet: Removed LNI 192.168.201.22@tcp [ 851.918957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.963705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 851.980229] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 860.670862] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:27:38 (1789518458) [ 860.843790] LNet: 48039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.853212] LNetError: 48039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.875277] LNet: Removed LNI 192.168.201.22@tcp [ 861.656764] Key type .llcrypt unregistered [ 861.659183] Key type ._llcrypt unregistered [ 869.978879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 870.228757] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 870.237191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 870.241860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 872.174485] Key type ._llcrypt registered [ 872.178825] Key type .llcrypt registered [ 872.438794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.457196] alg: No test for adler32 (adler32-zlib) [ 873.426624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.503781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.536553] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 874.553263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.583156] LNet: 48480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.588302] LNetError: 48480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.603931] LNet: Removed LNI 192.168.201.22@tcp [ 874.622509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.660856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 874.675651] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 881.423314] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:27:59 (1789518479) [ 881.534974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.565881] LNet: 49070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.576383] LNetError: 49070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.594087] LNet: Removed LNI 192.168.201.22@tcp [ 881.630753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.769475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 890.127897] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:28:07 (1789518487) [ 890.216186] LNet: 49652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.055834] Key type .llcrypt unregistered [ 891.063096] Key type ._llcrypt unregistered [ 898.767920] Key type ._llcrypt registered [ 898.776734] Key type .llcrypt registered [ 898.990636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 899.011401] alg: No test for adler32 (adler32-zlib) [ 899.971873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.026758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.054444] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 901.098753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.128591] LNet: 50056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.135525] LNetError: 50056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.145680] LNet: Removed LNI 192.168.201.22@tcp [ 901.159243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.263603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 901.294386] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 907.853269] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:28:26 (1789518506) [ 907.990575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.009637] LNet: 50650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.015846] LNetError: 50650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.048237] LNet: Removed LNI 192.168.201.22@tcp [ 908.068557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.110166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 915.510379] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:28:33 (1789518513) [ 917.204530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 919.165713] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:28:36 (1789518516) [ 920.633348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 922.398669] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:28:40 (1789518520) [ 924.013661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 926.243859] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:28:43 (1789518523) [ 927.989511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 929.659165] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:28:47 (1789518527) [ 931.043172] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 932.243602] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:28:50 (1789518530) [ 932.534799] LNet: 52674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.033048] Key type .llcrypt unregistered [ 933.037075] Key type ._llcrypt unregistered [ 939.256605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 939.550981] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 939.557349] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 939.562671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.253832] Key type ._llcrypt registered [ 942.261433] Key type .llcrypt registered [ 942.488563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.503279] alg: No test for adler32 (adler32-zlib) [ 943.393344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.443753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 943.462772] LNet: Added LNI 192.168.201.22@tcp [8/512/0/180] [ 943.465770] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 943.467352] LNet: 52992:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 943.548633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.564446] LNet: 53011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.569740] LNetError: 53011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 943.574372] LNet: Removed LNI 10.1.2.3@tcp [ 943.576590] LNetError: 53011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.587730] LNet: Removed LNI 192.168.201.22@tcp [ 943.615855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.664785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 943.680978] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 943.734782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.749668] LNet: 53035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.752716] LNetError: 53035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.759627] LNet: Removed LNI 192.168.201.22@tcp [ 943.768725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.800676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 943.824053] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 943.828335] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 943.831464] LNet: 53043:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 943.902140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.921845] LNet: 53061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.927786] LNetError: 53061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 943.934326] LNet: Removed LNI 10.1.2.3@tcp [ 943.940886] LNetError: 53061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.952872] LNet: Removed LNI 192.168.201.22@tcp [ 943.967596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.039682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.093096] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 944.104054] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.106628] LNet: 53069:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.277559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.292658] LNet: 53095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.297834] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.303188] LNet: Removed LNI 10.1.2.3@tcp [ 944.305448] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.314901] LNet: Removed LNI 192.168.201.22@tcp [ 944.344611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.403126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.427119] LNet: Added LNI 192.168.201.22@tcp [8/256/0/182] [ 944.433502] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 944.526459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.547136] LNet: 53123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.554376] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 944.564200] LNet: Removed LNI 10.1.2.3@tcp [ 944.566697] LNetError: 53123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.577987] LNet: Removed LNI 192.168.201.22@tcp [ 944.590966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.673474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 944.688825] LNet: Added LNI 192.168.201.22@tcp [8/256/0/182] [ 944.691240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 944.692597] LNet: 53131:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 944.724459] LNet: Removed LNI 10.1.2.3@tcp [ 944.736650] LNet: Removed LNI 192.168.201.22@tcp [ 945.190311] Key type .llcrypt unregistered [ 945.197111] Key type ._llcrypt unregistered [ 954.996364] Key type ._llcrypt registered [ 954.998811] Key type .llcrypt registered [ 956.103378] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 20:29:14 (1789518554) [ 957.723259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 959.609632] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:29:17 (1789518557) [ 960.156699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 960.185523] alg: No test for adler32 (adler32-zlib) [ 961.287830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.364206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.400054] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 962.450850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.22 [ 962.495250] LNet: 54393:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 962.502362] LNet: Removed LNI 192.168.201.22@tcp [ 970.391163] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:29:28 (1789518568) [ 970.569272] LNet: 54957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.584811] LNetError: 54957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.607283] LNet: Removed LNI 192.168.201.22@tcp [ 971.570785] Key type .llcrypt unregistered [ 971.577358] Key type ._llcrypt unregistered [ 979.114777] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.586141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 982.565250] Key type ._llcrypt registered [ 982.570268] Key type .llcrypt registered [ 982.733364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.753462] alg: No test for adler32 (adler32-zlib) [ 984.386096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 991.413760] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:29:49 (1789518589) [ 991.567038] LNet: 55875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.572447] LNetError: 55875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.589634] LNet: Removed LNI 10.1.2.3@tcp [ 992.229762] Key type .llcrypt unregistered [ 992.232836] Key type ._llcrypt unregistered [ 999.636233] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1000.082297] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.090911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.098528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1003.185495] Key type ._llcrypt registered [ 1003.187940] Key type .llcrypt registered [ 1003.438746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1003.472714] alg: No test for adler32 (adler32-zlib) [ 1004.651585] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1012.464597] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:30:10 (1789518610) [ 1014.645561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1016.109552] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:30:14 (1789518614) [ 1016.457988] LNet: 57059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.470881] LNetError: 57059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1016.485496] LNet: Removed LNI 10.1.2.3@tcp [ 1017.289169] Key type .llcrypt unregistered [ 1017.295070] Key type ._llcrypt unregistered [ 1024.340905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.815991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1024.826698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1024.834260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1027.864281] Key type ._llcrypt registered [ 1027.867173] Key type .llcrypt registered [ 1028.156152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.183395] alg: No test for adler32 (adler32-zlib) [ 1029.074812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.144469] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1039.542320] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:30:36 (1789518636) [ 1040.055144] LNet: 58287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.065563] LNetError: 58287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1040.083704] LNet: Removed LNI 10.1.2.3@tcp [ 1040.868888] Key type .llcrypt unregistered [ 1040.871385] Key type ._llcrypt unregistered [ 1047.806404] Key type ._llcrypt registered [ 1047.812840] Key type .llcrypt registered [ 1048.026486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.038904] alg: No test for adler32 (adler32-zlib) [ 1049.000732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1049.025974] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1057.048499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1057.997874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 1058.030473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 1058.087398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1058.131174] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1058.277219] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1058.373589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.398667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1058.420913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.453897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.485520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1058.512728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.538669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.562426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1058.588571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1058.603878] LNet: 58724:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.617807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1058.639198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1058.667600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1058.697974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1058.724690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.758528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1058.786096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.208524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 1059.249613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 1059.288649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1059.321831] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1059.562641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.608102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1059.653771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.703303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.751958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1059.789873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.827855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.881265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.925202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.939426] LNet: 58835:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.963587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.012795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.055633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.104762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.150422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.191868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.247671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1060.938634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 1060.969698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 1061.007137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1061.037946] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1061.331471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.353613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.377674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.405445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.438137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1061.463847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.508643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.583354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.623954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.648743] LNet: 58946:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.663752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.694718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.748553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.790758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.834815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.884103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.940549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1062.752434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 1062.808330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 1062.853935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1063.228393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.263077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.284191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.328832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.370756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1063.394684] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1063.399925] LNet: Skipped 1 previous similar message [ 1063.417659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.448611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.485486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.527201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.565649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.612190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.659372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.692158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.747487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.800302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.874873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.432297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 1064.474768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 1064.526354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1064.774638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.796180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.824596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.851842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.875770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1064.898787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.926786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.949637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.976697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.996462] LNet: 59169:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.001198] LNet: 59169:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1065.016340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.040484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.068467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.096672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.126422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.154429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.186320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1065.675335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 1065.770249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 1065.821928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1066.094478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.123932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.150167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.176198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.227405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1066.281693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.316539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.372797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.413705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.438065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.502283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.544623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.573621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.602924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.633118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.678728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.996255] LNet: 59321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.001232] LNetError: 59321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.073419] LNet: Removed LNI 192.168.201.22@tcp [ 1068.676539] Key type .llcrypt unregistered [ 1068.684184] Key type ._llcrypt unregistered [ 1087.253761] Key type ._llcrypt registered [ 1087.255950] Key type .llcrypt registered [ 1088.404580] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:31:26 (1789518686) [ 1089.203522] Key type .llcrypt unregistered [ 1089.206955] Key type ._llcrypt unregistered [ 1094.572575] Key type ._llcrypt registered [ 1094.573934] Key type .llcrypt registered [ 1094.763092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.775620] alg: No test for adler32 (adler32-zlib) [ 1095.820052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1095.847042] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1104.410690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1116.483576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1116.530887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1116.589383] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1118.535733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 1118.586776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 1118.641508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 1118.682532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 1118.737572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 1118.803502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 1118.878756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 1118.939323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 1119.004169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1119.037920] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1119.050109] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1119.471749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.516592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.571533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.605701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.642871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1119.697525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.779947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.818815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.850111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.865317] LNet: 60791:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.902948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.934678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.990651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.048746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.097973] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1120.110049] LNet: Skipped 3 previous similar messages [ 1120.120931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.189470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.235469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.883334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 1120.951982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 1120.999440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 1121.031691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 1121.076836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 1121.132841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 1121.196739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 1121.241294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 1121.284626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1121.312675] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1121.577691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.605406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.637379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.656708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.683699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1121.704498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.742847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.792785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.820192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.856944] LNet: 60914:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.892546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.933050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.964453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.989379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.011579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.064356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.149716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1122.757536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 1122.794512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 1122.831086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 1122.866799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 1122.898629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 1122.934348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 1122.970962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 1123.013532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 1123.049195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1123.344282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.395536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.443282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.476416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.513288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1123.566308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.612603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.651636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.692635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.713911] LNet: 61037:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.732624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.759239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.791219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.825879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.853333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.894335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.935680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.555445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 1124.599538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 1124.638895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 1124.680843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 1124.714212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 1124.757657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 1124.821622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 1124.882692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 1124.942707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1124.967511] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1124.970330] LNet: Skipped 5 previous similar messages [ 1125.219714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.251427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.282708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.315929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.343530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1125.380637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.398330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.415476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.443970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.483410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.517733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.550865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.589845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.631986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.699754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.728352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.287755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 1126.306705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 1126.335838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 1126.374924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 1126.415997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 1126.448934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 1126.486471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 1126.547713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 1126.617273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1126.788413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.803642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.827375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.847394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.870942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1126.899358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.935716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.997952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.058601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.092403] LNet: 61284:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.100028] LNet: 61284:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1127.121556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.155839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.189495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.231766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.260627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.310595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.363961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.052614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 1128.092228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 1128.120155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 1128.148515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 1128.181537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 1128.212520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 1128.260470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 1128.300596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 1128.347327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1128.607621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.650321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.687601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.728258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.767635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1128.807447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.866145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.904104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.942235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.975321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.008549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.038987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.083175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.140438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.188583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.227797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.838963] LNet: 61483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.850911] LNetError: 61483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.860800] LNet: Removed LNI 192.168.201.22@tcp [ 1135.912931] LNet: Removed LNI 192.168.201.22@tcp1 [ 1136.616789] Key type .llcrypt unregistered [ 1136.620337] Key type ._llcrypt unregistered [ 1154.755779] Key type ._llcrypt registered [ 1154.757900] Key type .llcrypt registered [ 1156.265734] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:32:33 (1789518753) [ 1156.945372] Key type .llcrypt unregistered [ 1156.947515] Key type ._llcrypt unregistered [ 1163.896636] Key type ._llcrypt registered [ 1163.899401] Key type .llcrypt registered [ 1164.162699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.201527] alg: No test for adler32 (adler32-zlib) [ 1165.224719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1165.278895] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1173.581981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1174.222901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1174.253450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1174.281614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1174.315074] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1174.600793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.629438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.691588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.716799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.747621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1174.777790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.797533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.823360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.850665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.869786] LNet: 62730:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.883872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.923322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.998809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.068845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.145640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.220699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.288541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.126725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1176.199244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1176.281777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1176.326542] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1176.756670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.830435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.881763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.953771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.010048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1177.073252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.139450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.215605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.277703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.329034] LNet: 62842:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.360288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.428544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.470697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.506762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.573814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.630826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.687820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.664559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1178.713730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1178.804580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1178.848323] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1179.115765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.162275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.241440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.329515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.370718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1179.424517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.464311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.509470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.555823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.583933] LNet: 62953:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.618759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.663506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.719627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.799657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.856595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.909935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.965765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1180.489580] LNet: 62994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.501910] LNetError: 62994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.522479] LNet: Removed LNI 192.168.201.22@tcp [ 1181.363135] Key type .llcrypt unregistered [ 1181.370100] Key type ._llcrypt unregistered [ 1204.310740] Key type ._llcrypt registered [ 1204.314565] Key type .llcrypt registered [ 1205.744964] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:33:23 (1789518803) [ 1206.595246] Key type .llcrypt unregistered [ 1206.599214] Key type ._llcrypt unregistered [ 1213.618338] Key type ._llcrypt registered [ 1213.620413] Key type .llcrypt registered [ 1213.838402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.852310] alg: No test for adler32 (adler32-zlib) [ 1214.763303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.808508] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1223.940722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1235.909663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.011787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.041588] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1237.737813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1237.791922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 1237.832022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1237.880374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 1237.918709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 1237.949341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 1238.006977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 1238.052278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 1238.094534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1238.143432] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1238.157088] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1238.366366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.405279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.438715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.470521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.500518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1238.529478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.556972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.587947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.617889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.629380] LNet: 64459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.643843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.670238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.697977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.715901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.770551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.825138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.880919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.933300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.630877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1239.668468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 1239.694955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1239.720582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 1239.796631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 1239.839203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 1239.924443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 1239.964488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 1240.033234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1240.063092] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1240.069121] LNet: Skipped 2 previous similar messages [ 1240.304370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.326676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.351977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.383221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.406497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1240.445624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.491456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.540265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.574405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.588534] LNet: 64584:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.607953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.646408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.675842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.712469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.758911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.811641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.838321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.897243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.632660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1241.695754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 1241.740275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1241.781860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 1241.809891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 1241.878553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 1241.953385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 1241.983474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 1242.026820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 1242.065456] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1242.311036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.352902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.394132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.446198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.494631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1242.549484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.588437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.632783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.685111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.717089] LNet: 64710:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.739096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.777286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.818912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.863726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.898539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.948289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.990896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.040645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.974815] LNet: 64787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.988490] LNetError: 64787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.993548] LNet: Removed LNI 192.168.201.22@tcp [ 1249.005114] LNet: Removed LNI 192.168.201.22@tcp1 [ 1249.434530] Key type .llcrypt unregistered [ 1249.439167] Key type ._llcrypt unregistered [ 1269.515802] Key type ._llcrypt registered [ 1269.518350] Key type .llcrypt registered [ 1271.686964] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:34:28 (1789518868) [ 1273.674855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1275.860544] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:34:33 (1789518873) [ 1277.067371] Key type .llcrypt unregistered [ 1277.071230] Key type ._llcrypt unregistered [ 1284.786143] Key type ._llcrypt registered [ 1284.790527] Key type .llcrypt registered [ 1285.154300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1285.228333] alg: No test for adler32 (adler32-zlib) [ 1286.166719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.222793] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1295.052408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1295.706508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1295.731670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1295.756976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1295.776279] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1295.980446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.006457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1296.032689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.072484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.111829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1296.142605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.175739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.210091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.260597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.289814] LNet: 66334:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.310531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.347317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.377352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.413740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.479614] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1296.483302] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1296.486331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.521079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.810518] LNet: 66373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.816362] LNetError: 66373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.826925] LNet: Removed LNI 192.168.201.22@tcp [ 1297.231504] Key type .llcrypt unregistered [ 1297.234868] Key type ._llcrypt unregistered [ 1317.038686] Key type ._llcrypt registered [ 1317.045246] Key type .llcrypt registered [ 1317.353840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.376691] alg: No test for adler32 (adler32-zlib) [ 1318.318929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1318.342813] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1327.176679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1338.272789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.305741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1338.327845] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1340.588882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1340.678929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 1340.740267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1340.813315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 1340.874256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 1340.986255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 1341.068456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 1341.123887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 1341.187220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1341.233380] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1341.542958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.564402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1341.580492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.602730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.616335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1341.634860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.652224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.673990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.702665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.717499] LNet: 67249:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.727742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.752335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.799514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.889567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.919158] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1341.929391] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1341.946639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.018768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.795950] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1349.225824] LNet: 67323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.237264] LNetError: 67323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.249890] LNet: Removed LNI 192.168.201.22@tcp [ 1349.263729] LNet: Removed LNI 192.168.201.22@tcp1 [ 1349.732500] Key type .llcrypt unregistered [ 1349.734257] Key type ._llcrypt unregistered [ 1370.909763] Key type ._llcrypt registered [ 1370.913779] Key type .llcrypt registered [ 1372.534102] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:36:10 (1789518970) [ 1373.130842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1373.157888] alg: No test for adler32 (adler32-zlib) [ 1374.172880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.245710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.285862] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1375.408243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.428479] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1376.971671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1377.083682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1377.148652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1377.212241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1377.262098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1377.324503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1377.380943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1377.402289] LNetError: 68424:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1378.336420] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1378.352033] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1380.447619] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1381.168432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1384.544840] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1384.830221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1384.883284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1384.991768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.016097] LNet: 68531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.022126] LNetError: 68531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.026990] LNet: Removed LNI 192.168.201.22@tcp [ 1385.034183] LNet: Removed LNI 192.168.201.22@tcp1 [ 1385.070650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.177320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.203347] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1386.495878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1386.546940] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1387.678827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1387.728525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1387.841809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1387.975995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1388.067378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1388.172642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1388.282925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1388.379490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1388.423064] LNetError: 68804:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1389.216885] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1392.377971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1395.368467] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1395.381965] LNet: Skipped 1 previous similar message [ 1400.795931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1400.856437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1400.934476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.176837] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:36:46 (1789519006) [ 1409.608492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.641620] LNet: 69523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.649256] LNetError: 69523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.652295] LNetError: 69523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1409.670706] LNet: Removed LNI 192.168.201.22@tcp [ 1409.689235] LNet: Removed LNI 192.168.201.22@tcp1 [ 1409.747658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.831886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.856523] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1410.517414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1410.554462] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1411.855266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1411.976965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1412.090551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1412.162099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1412.211420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.284697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.335540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 1412.473199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1412.831150] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.22@tcp [ 1413.857932] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1413.882611] LNet: Skipped 1 previous similar message [ 1428.246708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.418088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1428.449745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.480657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 500 [ 1428.579819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.412665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.543072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.584137] LNet: 70014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.591675] LNetError: 70014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.596267] LNetError: 70014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1435.602248] LNet: Removed LNI 192.168.201.22@tcp [ 1435.611435] LNet: Removed LNI 192.168.201.22@tcp1 [ 1435.631963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.687691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.743055] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1436.439488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.460338] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1437.135529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1437.170699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.222769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1437.243663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1437.271891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1437.295522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1437.321585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.348473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 1438.751395] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1438.756267] LNet: Skipped 3 previous similar messages [ 1439.925380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1449.643960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1449.727827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1449.824169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1460.701255] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:37:38 (1789519058) [ 1461.823056] LNet: 71063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.842627] LNetError: 71063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.850364] LNetError: 71063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1461.877346] LNet: Removed LNI 192.168.201.22@tcp [ 1461.897417] LNet: Removed LNI 192.168.201.22@tcp1 [ 1462.796848] Key type .llcrypt unregistered [ 1462.799981] Key type ._llcrypt unregistered [ 1470.617080] Key type ._llcrypt registered [ 1470.623798] Key type .llcrypt registered [ 1470.924446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.952791] alg: No test for adler32 (adler32-zlib) [ 1472.014463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1472.104674] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1482.190722] Lustre: DEBUG MARKER: Initial discovery [ 1482.753668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1486.632233] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1487.187437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1487.250827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1487.317488] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1487.350580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.428725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -l 3 [ 1487.510753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -e local_error [ 1487.537512] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1487.618278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m PUT -l 6 [ 1490.211711] Lustre: DEBUG MARKER: Force oleg122-client.virtnet to discover 192.168.201.122@tcp (in background) [ 1490.391505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 1492.773602] Lustre: DEBUG MARKER: Force oleg122-server to discover 192.168.201.22@tcp [ 1493.536733] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1497.269325] Lustre: DEBUG MARKER: Wait for 71817 [ 1497.832790] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1499.851495] Lustre: DEBUG MARKER: Finished wait on 71817 [ 1500.128555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1500.208580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1500.592041] LNet: 72276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.600571] LNetError: 72276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.684228] LNet: Removed LNI 192.168.201.22@tcp [ 1502.532672] Key type .llcrypt unregistered [ 1502.547085] Key type ._llcrypt unregistered [ 1528.691377] Key type ._llcrypt registered [ 1528.692946] Key type .llcrypt registered [ 1530.513417] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:38:47 (1789519127) [ 1531.087328] Key type .llcrypt unregistered [ 1531.088963] Key type ._llcrypt unregistered [ 1538.964461] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1539.237979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1539.253699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1539.263216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1542.217323] Key type ._llcrypt registered [ 1542.223220] Key type .llcrypt registered [ 1542.457806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.470292] alg: No test for adler32 (adler32-zlib) [ 1543.402167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.499147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.528181] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1544.728188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1544.755328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1556.077197] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:39:13 (1789519153) [ 1556.162560] LNet: 74344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.176793] LNetError: 74344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1556.182508] LNet: Removed LNI 10.1.2.3@tcp [ 1556.195697] LNet: Removed LNI 192.168.201.22@tcp [ 1557.322565] Key type .llcrypt unregistered [ 1557.324381] Key type ._llcrypt unregistered [ 1567.670683] Key type ._llcrypt registered [ 1567.679845] Key type .llcrypt registered [ 1568.152746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1568.188992] alg: No test for adler32 (adler32-zlib) [ 1569.302283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.354938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.376092] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1570.162361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1570.210616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1571.372764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.22@tcp [ 1572.451778] LNetError: Unexpected error -22 connecting to 192.168.201.22@tcp at host 192.168.201.22:988 [ 1579.421741] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:39:37 (1789519177) [ 1579.527838] LNet: 75473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.542789] LNetError: 75473:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1579.553949] LNet: Removed LNI 10.1.2.3@tcp [ 1579.583531] LNet: Removed LNI 192.168.201.22@tcp [ 1580.329436] Key type .llcrypt unregistered [ 1580.332925] Key type ._llcrypt unregistered [ 1586.578319] Key type ._llcrypt registered [ 1586.583615] Key type .llcrypt registered [ 1586.778984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1586.792446] alg: No test for adler32 (adler32-zlib) [ 1587.773177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.833845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1587.862777] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1588.817818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1588.839720] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1589.582147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.22@tcp1 --nid 192.168.201.22@tcp2 [ 1593.822500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp2 [ 1593.844359] LNetError: 76314:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.22@tcp1 to 192.168.201.22@tcp2 - Route table may be misconfigured [ 1593.869884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp2 192.168.201.22@tcp1 [ 1602.175637] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:39:59 (1789519199) [ 1602.278815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.320607] LNet: 76882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.327620] LNetError: 76882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.359213] LNet: Removed LNI 192.168.201.22@tcp1 [ 1603.368849] LNetError: 76882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.408412] LNet: Removed LNI 192.168.201.22@tcp2 [ 1603.467574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.549559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.593177] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1604.297758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1604.312854] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1605.012418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1605.046332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.22@tcp [ 1605.084969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp1 [ 1605.117783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp -e network_timeout [ 1605.174796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -e network_timeout [ 1605.203452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -e network_timeout [ 1605.237936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -e network_timeout [ 1605.269746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1605.295195] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1605.323800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1605.442512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1605.610466] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1605.630049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1611.955527] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:40:10 (1789519210) [ 1612.024265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.058306] LNet: 77742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.063291] LNetError: 77742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.070600] LNet: Removed LNI 192.168.201.22@tcp [ 1613.096090] LNet: Removed LNI 192.168.201.22@tcp1 [ 1613.133231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.234457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1613.628662] LNet: 77786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.084383] Key type .llcrypt unregistered [ 1614.086379] Key type ._llcrypt unregistered [ 1626.657561] Key type ._llcrypt registered [ 1626.660658] Key type .llcrypt registered [ 1628.251624] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:40:25 (1789519225) [ 1628.690717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.706461] alg: No test for adler32 (adler32-zlib) [ 1629.734106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.333465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1633.493268] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:40:31 (1789519231) [ 1633.985802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.016941] LNet: 78883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.057164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.126430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.165923] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1634.947507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1634.964273] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1635.671469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1635.703760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp1 [ 1635.731511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp1 [ 1643.223341] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:40:40 (1789519240) [ 1645.046815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg122-server [ 1647.217375] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:40:44 (1789519244) [ 1649.135622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg122-server [ 1650.841871] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:40:48 (1789519248) [ 1652.539850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg122-server [ 1654.443423] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:40:52 (1789519252) [ 1656.680212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg122-server [ 1658.348289] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:40:56 (1789519256) [ 1660.012979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg122-server [ 1661.899828] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:40:59 (1789519259) [ 1664.280340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg122-server [ 1666.579900] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:41:04 (1789519264) [ 1668.719970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg122-server [ 1670.418413] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:41:08 (1789519268) [ 1671.972451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg122-server [ 1673.962768] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:41:11 (1789519271) [ 1676.381392] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg122-server [ 1677.843147] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:41:16 (1789519276) [ 1678.112967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.152384] LNet: 82761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.166879] LNetError: 82761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.173629] LNet: Removed LNI 192.168.201.22@tcp [ 1679.210760] LNet: Removed LNI 192.168.201.22@tcp1 [ 1679.274223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.370683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.421607] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1680.310111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1680.425473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1680.651947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.681161] LNet: 82908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.685894] LNetError: 82908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.688757] LNetError: 82908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1680.723574] LNet: Removed LNI 192.168.201.22@tcp [ 1680.745040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.861748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1680.901648] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1681.848602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1682.109458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.139239] LNetError: 83053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.182667] LNet: Removed LNI 192.168.201.22@tcp [ 1683.249610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.386949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.461793] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1684.085722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1684.155957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1684.291700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.309563] LNet: 83201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.314943] LNet: 83201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1684.322407] LNetError: 83201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.344897] LNet: Removed LNI 192.168.201.22@tcp [ 1684.372847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.422665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1684.445666] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1685.226670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1685.438866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.507085] LNet: Removed LNI 192.168.201.22@tcp [ 1686.556209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.663789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.700324] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1687.745041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1687.819238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1687.980547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.063907] LNet: Removed LNI 192.168.201.22@tcp [ 1689.109574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.212877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1689.251370] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1690.153928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1690.509654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.544710] LNet: 83639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.554918] LNet: 83639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1690.573962] LNetError: 83639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.584408] LNetError: 83639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1690.614762] LNet: Removed LNI 192.168.201.22@tcp [ 1690.647436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.772894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.829288] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1691.454455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1691.532447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1691.730694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.790029] LNet: Removed LNI 192.168.201.22@tcp [ 1692.809444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.908010] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1692.938925] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1693.721303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1695.299606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.370914] LNet: Removed LNI 192.168.201.22@tcp [ 1696.446928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.601285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.674109] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1697.679617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1697.759355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1698.074600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.111202] LNet: Removed LNI 192.168.201.22@tcp [ 1699.184510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.280943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1699.332914] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1700.206770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1700.520857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.542832] LNet: 84234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.548840] LNet: 84234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1700.555941] LNetError: 84234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.560762] LNetError: 84234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1701.606268] LNet: Removed LNI 192.168.201.22@tcp [ 1701.727178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.834392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.874705] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1702.732775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1702.832535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1703.266960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.360185] LNet: Removed LNI 192.168.201.22@tcp [ 1704.403525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.488199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1704.521137] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1705.963268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1706.388600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.561515] LNet: Removed LNI 192.168.201.22@tcp [ 1707.603635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.758986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.809562] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1708.828422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1708.930524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.992189] LNet: Removed LNI 192.168.201.22@tcp [ 1709.043026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.142369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.182230] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1710.163951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1710.273176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.340893] LNet: Removed LNI 192.168.201.22@tcp [ 1710.377256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.473416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1710.554646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.588341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.646805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1710.757260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.795580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.868840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1711.049565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.133994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.247913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1711.300943] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1712.441572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.497291] LNet: Removed LNI 192.168.201.22@tcp [ 1712.547975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.691959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1720.354855] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:41:58 (1789519318) [ 1720.524280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.550239] LNet: 85544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.570276] LNet: 85544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1720.603940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.701049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1720.746040] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1722.279425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.322042] LNetError: 85687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.329818] LNetError: 85687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1722.354795] LNet: Removed LNI 192.168.201.22@tcp231 [ 1722.389788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.473237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1722.487420] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1722.491234] LNet: 85695:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.111341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1723.149291] LNet: Removed LNI 192.168.201.22@tcp231 [ 1723.162646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1723.198469] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1724.440560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.462215] LNet: Removed LNI 192.168.201.22@tcp231 [ 1724.476533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.533283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1724.559935] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 1724.567604] LNet: 85847:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1725.037744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1725.067870] LNet: Removed LNI 192.168.201.22@tcp231 [ 1725.079799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1725.103935] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 1726.262564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.293991] LNet: Removed LNI 192.168.201.22@tcp231 [ 1726.313312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.373800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1726.393861] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 1726.400234] LNet: 85999:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1726.991536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.031774] LNet: Removed LNI 192.168.201.22@tcp231 [ 1727.052512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1727.094934] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 1728.181827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.210476] LNet: Removed LNI 192.168.201.22@tcp231 [ 1728.237198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.317527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1728.350582] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1728.847646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1728.874661] LNet: Removed LNI 192.168.201.22@tcp231 [ 1728.888868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1728.919526] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 1738.701712] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:42:16 (1789519336) [ 1738.863722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.928946] LNet: Removed LNI 192.168.201.22@tcp231 [ 1738.951477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.035572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1739.056140] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1740.141569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1740.206483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1740.249533] LNet: Removed LNI 192.168.201.22@tcp232 [ 1740.267666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1740.314486] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1741.512993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1741.550734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1741.569938] LNet: Removed LNI 192.168.201.22@tcp232 [ 1741.592828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1741.620336] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1742.443418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1742.473239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1742.502662] LNet: Removed LNI 192.168.201.22@tcp232 [ 1742.516081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1742.566210] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 1743.274973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1743.315484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1743.351650] LNet: Removed LNI 192.168.201.22@tcp232 [ 1743.362582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1749.644131] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:42:27 (1789519347) [ 1751.094945] Key type .llcrypt unregistered [ 1751.102343] Key type ._llcrypt unregistered [ 1759.397898] Key type ._llcrypt registered [ 1759.403220] Key type .llcrypt registered [ 1759.710829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1759.723240] alg: No test for adler32 (adler32-zlib) [ 1760.719877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1760.752480] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1769.567776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1779.841900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.957980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1779.995053] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1781.681833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 1781.730407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 1781.782955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 1781.849394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 1781.893819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 1781.952595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 1781.983507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 1782.026328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 1782.075966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1782.100865] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1782.104488] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1782.386274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1782.418645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1782.461770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1782.484759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1782.518663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1782.563814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1782.603576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1782.640869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.692700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1782.715480] LNet: 88653:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.738703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1782.769720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1782.801048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1782.837929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.872791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.895430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1783.312500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 1783.349908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 1783.392542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 1783.425284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 1783.456359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 1783.483705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 1783.518442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 1783.550937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 1783.624261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1783.675494] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1784.011457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1784.087629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1784.139258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1784.205791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1784.261876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1784.311605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1784.355603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1784.408904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1784.453866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1784.481288] LNet: 88775:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.514113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1784.542853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1784.584669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1784.651467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1784.690139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1784.737224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1785.479411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 1785.543317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 1785.581856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 1785.623550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 1785.675906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 1785.720411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 1785.801258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 1785.847500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 1785.898457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1785.919428] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1786.191473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1786.242332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1786.277363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1786.325422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1786.360405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1786.397447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1786.425870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1786.457536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1786.504726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1786.540347] LNet: 88898:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1786.563869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1786.600310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1786.641982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1786.709787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1786.791657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1786.831073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1787.242682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 1787.267893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 1787.300556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 1787.330661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 1787.355300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 1787.380879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 1787.410683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 1787.448358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 1787.488992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1787.786189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1787.809821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1787.835411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1787.860651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1787.885639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1787.908513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1787.938061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1787.973858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1788.001515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1788.027781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1788.092868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1788.149689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1788.210682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1788.266715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1788.333306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1788.861664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 1788.897989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 1788.947254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 1788.978219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 1789.011533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 1789.047802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 1789.088478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 1789.131427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 1789.180840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1789.215259] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 1789.222084] LNet: Skipped 1 previous similar message [ 1789.446628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.474525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.504313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.534530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.558687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1789.580832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1789.602710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1789.627324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1789.649548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1789.668740] LNet: 89142:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1789.674401] LNet: 89142:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1789.696345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1789.725400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1789.755957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1789.780162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1789.806182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1789.848489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.353113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 1790.385924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 1790.411711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 1790.437612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 1790.464668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 1790.490664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 1790.514555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 1790.538430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 1790.576654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 1790.595740] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1790.831909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1790.860512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1790.885370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1790.913665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1790.941768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1790.967519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1791.003664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1791.035917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1791.076462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1791.103188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1791.130959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1791.161652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1791.191404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1791.217127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1791.246755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1797.057311] LNet: 89339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.062270] LNetError: 89339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.078714] LNet: Removed LNI 192.168.201.22@tcp [ 1797.101418] LNet: Removed LNI 192.168.201.22@tcp1 [ 1797.577767] Key type .llcrypt unregistered [ 1797.579281] Key type ._llcrypt unregistered [ 1816.060324] Key type ._llcrypt registered [ 1816.062595] Key type .llcrypt registered [ 1817.559967] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:43:35 (1789519415) [ 1818.027979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.040887] alg: No test for adler32 (adler32-zlib) [ 1818.974318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.041159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1819.062478] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1819.786781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1819.842865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 1819.897679] LNetError: 90316:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1820.017864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.22@tcp [ 1820.165473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.237984] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:43:45 (1789519425) [ 1827.447068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.500851] LNet: 90906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.512631] LNetError: 90906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.545414] LNet: Removed LNI 192.168.201.22@tcp [ 1827.581709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.641921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.671780] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1828.396361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1828.417897] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1829.437660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.22@tcp [ 1829.461562] LNetError: 91172:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 1829.663374] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 1829.666810] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 1830.700741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.22@tcp [ 1839.227216] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:43:57 (1789519437) [ 1840.881212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg122-server [ 1842.461227] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:44:00 (1789519440) [ 1844.544040] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg122-server [ 1846.187498] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:44:04 (1789519444) [ 1847.996039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg122-server [ 1849.566474] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:44:07 (1789519447) [ 1849.901203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.926828] LNet: 92795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.937294] LNetError: 92795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.955115] LNet: Removed LNI 192.168.201.22@tcp [ 1849.971893] LNet: Removed LNI 192.168.201.22@tcp1 [ 1850.004057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.078600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.115173] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1850.828385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1850.912102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1850.962473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1851.074563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1851.136639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1851.219967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1858.609861] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:44:16 (1789519456) [ 1858.671280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.690561] LNet: 93521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.694860] LNetError: 93521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.702774] LNetError: 93521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1858.730524] LNet: Removed LNI 192.168.201.22@tcp [ 1858.774169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.830868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1858.865080] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1866.867370] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:44:24 (1789519464) [ 1867.048727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.120956] LNet: 94224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.134398] LNetError: 94224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.171527] LNet: Removed LNI 192.168.201.22@tcp [ 1867.207353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.292200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1867.332869] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1876.822442] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:44:34 (1789519474) [ 1879.380598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1881.580669] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:44:39 (1789519479) [ 1882.359971] LNet: 95266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.371581] LNetError: 95266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.396604] LNet: Removed LNI 192.168.201.22@tcp [ 1883.242728] Key type .llcrypt unregistered [ 1883.246740] Key type ._llcrypt unregistered [ 1893.559664] Key type ._llcrypt registered [ 1893.570572] Key type .llcrypt registered [ 1893.898397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.932273] alg: No test for adler32 (adler32-zlib) [ 1894.856642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1894.873568] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1902.870792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1909.863960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.122@tcp [ 1910.450569] LNet: 95683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.470472] LNetError: 95683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.501658] LNet: Removed LNI 192.168.201.22@tcp [ 1911.024452] Key type .llcrypt unregistered [ 1911.026971] Key type ._llcrypt unregistered [ 1924.017268] Key type ._llcrypt registered [ 1924.020186] Key type .llcrypt registered [ 1925.538212] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1927.736904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1929.726677] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:45:27 (1789519527) [ 1931.787989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1933.605778] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:45:31 (1789519531) [ 1936.483136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg122-server [ 1938.755587] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:45:36 (1789519536) [ 1941.180955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg122-server [ 1943.383083] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:45:41 (1789519541) [ 1945.496246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1945.576303] alg: No test for adler32 (adler32-zlib) [ 1947.010739] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 1947.513033] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1949.287200] Key type lgssc registered [ 1951.138862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1973.440495] Key type lgssc unregistered [ 1973.864304] LNet: 98855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.891480] LNetError: 98855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.933440] LNet: Removed LNI 192.168.201.22@tcp [ 1974.945735] Key type .llcrypt unregistered [ 1974.947229] Key type ._llcrypt unregistered [ 1991.594539] Key type ._llcrypt registered [ 1991.596594] Key type .llcrypt registered [ 1993.025136] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:46:30 (1789519590) [ 1993.490425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.506780] alg: No test for adler32 (adler32-zlib) [ 1994.490695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.610687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.700254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.761814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1994.874826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2002.890622] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:46:40 (1789519600) [ 2003.172647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.204907] LNet: 100269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.230805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.264118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2003.288611] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2004.197346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.220985] LNet: 100413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.229498] LNetError: 100413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.245762] LNet: Removed LNI 192.168.201.22@tcp [ 2004.261237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.322716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2004.345978] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2004.932899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.963597] LNetError: 100441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.976706] LNet: Removed LNI 192.168.201.22@tcp [ 2004.993500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.049961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2005.082639] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2012.058400] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:46:49 (1789519609) [ 2012.414168] LNet: 101032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.426328] LNet: 101032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2012.440955] LNetError: 101032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.472701] LNet: Removed LNI 192.168.201.22@tcp [ 2013.547905] Key type .llcrypt unregistered [ 2013.551868] Key type ._llcrypt unregistered [ 2022.487991] Key type ._llcrypt registered [ 2022.499066] Key type .llcrypt registered [ 2022.908690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2022.954813] alg: No test for adler32 (adler32-zlib) [ 2023.861580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.926963] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2023.929505] LNet: *** cfs_fail_loc=e003, val=100*** [ 2024.502737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2024.533518] LNet: *** cfs_fail_loc=e003, val=100*** [ 2024.538959] LNet: Skipped 2 previous similar messages [ 2024.547821] Ping reply received with mapping: [ 2024.549591] found metadata reply NI[192.168.201.22@tcp] [ 2024.552290] 16 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2024.555052] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2024.559477] 5a 5a 5a 5a 5a 5a 5a 5a [ 2024.698214] LNet: 101427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2024.703785] LNetError: 101427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2025.765336] LNet: Removed LNI 192.168.201.22@tcp [ 2026.381562] Key type .llcrypt unregistered [ 2026.385990] Key type ._llcrypt unregistered [ 2040.896454] Key type ._llcrypt registered [ 2040.900628] Key type .llcrypt registered [ 2042.806114] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:47:20 (1789519640) [ 2043.502691] Key type .llcrypt unregistered [ 2043.505489] Key type ._llcrypt unregistered [ 2050.765445] Key type ._llcrypt registered [ 2050.770039] Key type .llcrypt registered [ 2050.978066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.003074] alg: No test for adler32 (adler32-zlib) [ 2052.102241] LNetError: 102465:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2052.491558] Key type .llcrypt unregistered [ 2052.493306] Key type ._llcrypt unregistered [ 2058.320451] Key type ._llcrypt registered [ 2058.322955] Key type .llcrypt registered [ 2059.589757] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:47:37 (1789519657) [ 2060.477702] Key type .llcrypt unregistered [ 2060.481624] Key type ._llcrypt unregistered [ 2067.904869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2067.930787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2071.166991] Key type ._llcrypt registered [ 2071.169531] Key type .llcrypt registered [ 2071.485234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2071.511693] alg: No test for adler32 (adler32-zlib) [ 2072.680908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.834209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2086.840197] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:48:04 (1789519684) [ 2087.064380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.089657] LNet: 104236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.134018] LNetError: 104236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2087.154388] LNet: Removed LNI 10.1.2.3@tcp [ 2087.191848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.127522] Key type .llcrypt unregistered [ 2088.131841] Key type ._llcrypt unregistered [ 2095.256658] Key type ._llcrypt registered [ 2095.258095] Key type .llcrypt registered [ 2095.413922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2095.424996] alg: No test for adler32 (adler32-zlib) [ 2096.507383] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2096.579903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2096.777938] LNet: 104527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.794314] LNetError: 104527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.812769] LNet: Removed LNI 192.168.201.22@tcp [ 2097.445530] Key type .llcrypt unregistered [ 2097.448676] Key type ._llcrypt unregistered [ 2109.334567] Key type ._llcrypt registered [ 2109.336794] Key type .llcrypt registered [ 2110.900565] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:48:28 (1789519708) [ 2111.434377] Key type .llcrypt unregistered [ 2111.437898] Key type ._llcrypt unregistered [ 2117.978316] Key type ._llcrypt registered [ 2117.980381] Key type .llcrypt registered [ 2118.165991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2118.183881] alg: No test for adler32 (adler32-zlib) [ 2125.958617] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:48:43 (1789519723) [ 2126.127049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.174895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.276293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.314289] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2127.052670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.082679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2127.112752] LNet: Removed LNI 192.168.201.22@tcp [ 2127.426081] LNet: 106267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.856454] Key type .llcrypt unregistered [ 2127.858126] Key type ._llcrypt unregistered [ 2142.904356] Key type ._llcrypt registered [ 2142.909396] Key type .llcrypt registered [ 2144.232069] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:49:02 (1789519742) [ 2144.704350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2144.719063] alg: No test for adler32 (adler32-zlib) [ 2145.742183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.797979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.818143] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2146.421956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2146.510970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.22@tcp [ 2146.547884] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2146.558843] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2146.567567] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2146, 0, 0 [ 2146.573189] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2146.575364] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2146.579392] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2146.584371] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2146, 0, 0 [ 2146.588274] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2146.590485] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2146.593411] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2146.595676] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2146, 0, 2196 [ 2146.599185] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2146.600790] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2146.602751] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2146.604676] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2146, 0, 2196 [ 2146.608350] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2153.788255] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:49:11 (1789519751) [ 2154.393613] LNet: 107806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.398827] LNetError: 107806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.409877] LNet: Removed LNI 192.168.201.22@tcp [ 2155.082541] Key type .llcrypt unregistered [ 2155.084634] Key type ._llcrypt unregistered [ 2165.182900] Key type ._llcrypt registered [ 2165.188950] Key type .llcrypt registered [ 2165.650059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2165.694878] alg: No test for adler32 (adler32-zlib) [ 2166.674989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2166.699118] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2178.704388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 2188.190435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2188.236994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2188.286063] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 2190.168102] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2190.402218] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2190.413666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2190.444829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2192.854762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2192.895140] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2207.922754] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:50:05 (1789519805) [ 2208.288690] LNet: 109315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.305953] LNetError: 109315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.316427] LNet: Removed LNI 192.168.201.22@tcp [ 2209.375975] LNet: Removed LNI 192.168.201.22@tcp1 [ 2209.378535] LNetError: 109315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2209.385174] LNetError: 109315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2209.408965] LNet: Removed LNI 10.1.2.3@tcp99 [ 2210.012604] Key type .llcrypt unregistered [ 2210.014205] Key type ._llcrypt unregistered [ 2217.761428] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2217.766052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2220.573505] Key type ._llcrypt registered [ 2220.577739] Key type .llcrypt registered [ 2220.819200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2220.832647] alg: No test for adler32 (adler32-zlib) [ 2221.797365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.876374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.929250] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2222.661816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2222.681308] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2224.640840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2224.691378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2224.753215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2224.782934] LNetError: 110057:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2224.844698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2224.905501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2225.605683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2225.684707] LNet: 110077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.689669] LNetError: 110077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2225.693900] LNet: Removed LNI 10.1.2.3@tcp [ 2226.748594] LNet: Removed LNI 192.168.201.22@tcp [ 2227.655428] Key type .llcrypt unregistered [ 2227.657224] Key type ._llcrypt unregistered [ 2240.626864] Key type ._llcrypt registered [ 2240.629132] Key type .llcrypt registered [ 2241.722717] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:50:39 (1789519839) [ 2242.081251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.103486] alg: No test for adler32 (adler32-zlib) [ 2243.075084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.127594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2243.155131] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2243.702228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2243.782119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg122-client@tcp [ 2249.747450] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:50:47 (1789519847) [ 2249.850144] LNet: 111601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.858773] LNetError: 111601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.900663] LNet: Removed LNI 192.168.201.22@tcp [ 2250.436989] Key type .llcrypt unregistered [ 2250.440882] Key type ._llcrypt unregistered [ 2256.850477] Key type ._llcrypt registered [ 2256.852847] Key type .llcrypt registered [ 2257.139907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2257.170637] alg: No test for adler32 (adler32-zlib) [ 2258.173418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2258.206911] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2258.240299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2264.592257] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:51:02 (1789519862) [ 2266.101365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2268.621503] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:51:06 (1789519866) [ 2270.184403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2271.749438] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:51:09 (1789519869) [ 2272.209824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.251824] LNet: 113027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.256703] LNetError: 113027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.279250] LNet: Removed LNI 192.168.201.22@tcp [ 2272.306890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.409134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.441870] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2273.445581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2273.488740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2280.386762] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:51:18 (1789519878) [ 2280.736449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.770434] LNet: 113738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.779702] LNetError: 113738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.809196] LNet: Removed LNI 192.168.201.22@tcp [ 2280.854785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.930780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2280.990223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.666666] Key type .llcrypt unregistered [ 2281.669461] Key type ._llcrypt unregistered [ 2295.999142] Key type ._llcrypt registered [ 2296.002552] Key type .llcrypt registered [ 2297.507249] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:51:35 (1789519895) [ 2297.962931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.989902] alg: No test for adler32 (adler32-zlib) [ 2298.907512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.953925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.986665] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2299.627651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2299.693904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2306.256089] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:51:44 (1789519904) [ 2306.766781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.848173] LNet: 115290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.864467] LNetError: 115290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.910285] LNet: Removed LNI 192.168.201.22@tcp [ 2307.034663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.164416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2307.213928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2316.243487] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:51:53 (1789519913) [ 2316.579259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.673859] LNet: 115867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.740713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.856226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2317.455838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2317.536364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2317.647358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2318.174334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2318.256941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2318.395761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.462405] LNet: 115904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.516723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.598967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2319.145213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2319.213902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2319.303633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2320.095050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2320.186198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2327.805792] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:52:05 (1789519925) [ 2327.986813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.011171] LNet: 116500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.045940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.147402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.178413] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2328.978613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2328.996700] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2329.754190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2329.804434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2330.032897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.071331] LNetError: 116784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.080727] LNet: Removed LNI 192.168.201.22@tcp [ 2330.095577] LNet: Removed LNI 192.168.201.22@tcp2 [ 2330.154612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.267085] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2330.275845] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2330.280966] LNet: 116790:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.847596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2331.126155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.159232] LNetError: 116827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.167397] LNetError: 116827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2331.178714] LNet: Removed LNI 192.168.201.22@tcp [ 2331.204429] LNet: Removed LNI 192.168.201.22@tcp2 [ 2331.237471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.319776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.360255] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2332.405917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2332.429772] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2333.519554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2333.599496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2333.661954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2333.943836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.975195] LNet: 117112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.981526] LNet: 117112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2333.988555] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.993596] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2333.997925] LNet: Removed LNI 192.168.201.22@tcp [ 2335.016379] LNet: Removed LNI 192.168.201.22@tcp2 [ 2335.048984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.126414] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2335.132757] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2335.139334] LNet: 117118:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.739952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2335.982527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.009873] LNet: Removed LNI 192.168.201.22@tcp [ 2337.062893] LNet: Removed LNI 192.168.201.22@tcp2 [ 2337.103930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.205390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.237942] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2338.065600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.100760] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2339.111410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2339.166362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2339.425332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.445748] LNetError: 117434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.453322] LNetError: 117434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2340.517502] LNet: Removed LNI 192.168.201.22@tcp [ 2340.532728] LNet: Removed LNI 192.168.201.22@tcp2 [ 2340.589189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.664468] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2340.672727] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2340.681458] LNet: 117441:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2341.386944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2341.649817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.751254] LNet: Removed LNI 192.168.201.22@tcp [ 2342.778431] LNet: Removed LNI 192.168.201.22@tcp2 [ 2342.859879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.990442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2343.038490] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2343.915381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2343.932736] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2344.959038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2344.996976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2345.063787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2345.335510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.361424] LNet: 117762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.365754] LNet: 117762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2346.400060] LNet: Removed LNI 192.168.201.22@tcp [ 2347.430224] LNet: Removed LNI 192.168.201.22@tcp2 [ 2347.457174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.509713] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2347.513437] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2347.518317] LNet: 117769:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.977581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2348.180801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.203728] LNetError: 117806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.210380] LNetError: 117806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2348.217258] LNet: Removed LNI 192.168.201.22@tcp [ 2349.286402] LNet: Removed LNI 192.168.201.22@tcp2 [ 2349.328326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.423923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.453515] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2350.470967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.487084] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2351.396686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2351.430314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2351.512569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2351.859883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.927144] LNet: Removed LNI 192.168.201.22@tcp [ 2352.941502] LNet: Removed LNI 192.168.201.22@tcp2 [ 2353.023243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.109308] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2353.124958] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2353.129583] LNet: 118108:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.640297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2353.897434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.975243] LNet: Removed LNI 192.168.201.22@tcp [ 2354.986979] LNet: Removed LNI 192.168.201.22@tcp2 [ 2355.026173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.071516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.087148] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2355.805221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.824781] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2356.577544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2356.621572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2356.645780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2356.706769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2356.996416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.047346] LNet: Removed LNI 192.168.201.22@tcp [ 2359.093493] LNet: Removed LNI 192.168.201.22@tcp2 [ 2359.145159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2359.184517] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2359.189960] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2359.790503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2367.586826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2369.797896] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:52:47 (1789519967) [ 2369.883882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.919995] LNet: 119222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.930252] LNet: 119222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2369.941548] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.950826] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2369.958876] LNet: Removed LNI 192.168.201.22@tcp [ 2370.981741] LNet: Removed LNI 192.168.201.22@tcp2 [ 2371.021993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.497540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.542272] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2373.049193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2373.073666] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2374.986634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.026948] LNet: Removed LNI 10.1.2.3@tcp [ 2375.040550] LNet: Removed LNI 192.168.201.22@tcp [ 2375.086940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.212353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2375.268799] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2375.276620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2376.222426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2376.248915] LNet: Removed LNI 192.168.201.22@tcp [ 2376.271227] LNet: Removed LNI 10.1.2.3@tcp [ 2383.374346] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 20:53:01 (1789519981) [ 2383.565217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.646722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.750422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.778278] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2384.486616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2384.523687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2384.576647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.614916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.638033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.658790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.689522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.724936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.769607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.808337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.842839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2384.879249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2385.023822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2391.155723] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 20:53:08 (1789519988) [ 2391.395841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.496869] LNet: Removed LNI 192.168.201.22@tcp [ 2392.550828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.651361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.693745] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2393.358543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2393.387349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2393.417688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.453311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.509422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.539961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.572366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.598484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.625801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.677714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.740608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.781427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.870862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2393.911149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.945582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2393.978472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.012492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.043442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.071791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.105565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.142831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.172818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2394.203807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 2400.449794] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:53:18 (1789519998) [ 2400.772917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.835416] LNet: Removed LNI 192.168.201.22@tcp [ 2401.856408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.993527] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2402.193163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2402.198448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2402.201986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2404.701575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2404.776823] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2406.153202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2406.195644] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2408.458821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2408.500271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2408.584741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.22@tcp -m PUT -e local_timeout -r 1 [ 2408.637409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2408.701679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2408.710324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2408.985197] LNetError: 121663:0:(lib-move.c:4794:lnet_parse()) 192.168.201.22@tcp, src 192.168.201.22@tcp: Dropping PUT (error -108 looking up sender) [ 2410.015557] LNet: 122121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.018556] LNet: 122121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2410.022784] LNetError: 122121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2410.026599] LNetError: 122121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2410.030332] LNet: Removed LNI 10.1.2.3@tcp [ 2410.042692] LNet: Removed LNI 192.168.201.22@tcp [ 2410.362355] Key type .llcrypt unregistered [ 2410.364285] Key type ._llcrypt unregistered [ 2423.945324] Key type ._llcrypt registered [ 2423.947858] Key type .llcrypt registered [ 2426.008831] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:53:43 (1789520023) [ 2426.444554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2426.457649] alg: No test for adler32 (adler32-zlib) [ 2427.585658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2427.982818] LNet: 122971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2428.601704] Key type .llcrypt unregistered [ 2428.604810] Key type ._llcrypt unregistered [ 2436.340458] Key type ._llcrypt registered [ 2436.341725] Key type .llcrypt registered [ 2436.698886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2436.712788] alg: No test for adler32 (adler32-zlib) [ 2437.678496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2437.706845] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2444.931610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 2445.163685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2445.736321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 2445.920157] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 2446.013290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.055273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2446.082617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.112385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2446.149465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2446.198613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2446.246439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2446.276700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.319647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.337331] LNet: 123421:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.347826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.374458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.404448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.432534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.470955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2446.510781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2447.174232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2447.443020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2447.470854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2447.511428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2447.552199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2447.587666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2447.610417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2447.645359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2447.677693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2447.700984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2447.718996] LNet: 123540:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2447.738766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2447.767460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2447.795811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2447.823774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2447.857853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.906053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2448.391675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 2448.649087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.679922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.712644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.732932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.758510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2448.782846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.813037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.839448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.881315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.900346] LNet: 123659:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.914542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.934320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.950174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2448.966832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.993756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.012430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2449.465177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2449.648555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.671198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2449.717934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2449.786839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2449.822247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2449.865549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2449.902571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.936848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.959438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.996115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2450.065964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.126563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.172161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.225091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.278199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.499917] LNet: 123825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.503900] LNetError: 123825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2450.516876] LNet: Removed LNI 192.168.201.22@tcp [ 2450.751841] Key type .llcrypt unregistered [ 2450.753631] Key type ._llcrypt unregistered [ 2465.462584] Key type ._llcrypt registered [ 2465.463847] Key type .llcrypt registered [ 2466.869920] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:54:24 (1789520064) [ 2467.401294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2467.447934] alg: No test for adler32 (adler32-zlib) [ 2468.383266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.815203] LNet: 124709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.364458] Key type .llcrypt unregistered [ 2469.367345] Key type ._llcrypt unregistered [ 2475.885491] Key type ._llcrypt registered [ 2475.887772] Key type .llcrypt registered [ 2476.033909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2476.046660] alg: No test for adler32 (adler32-zlib) [ 2476.957201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2476.980642] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2485.267901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 2494.927869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.964912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2494.991991] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 2496.129885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2496.608787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 2496.823757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2496.841902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2496.890578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2496.913816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2496.938656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2496.964807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2496.997826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2497.023134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2497.057243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2497.072822] LNet: 125359:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2497.107407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2497.166279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2497.214622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2497.240584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2497.256324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2497.278912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2497.997640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2498.248149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2498.278483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2498.322908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2498.369647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2498.415454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2498.461611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2498.495767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2498.532772] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.122@tcp, 192.168.201.122@tcp1 [ 2498.543240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2498.590645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2498.609241] LNet: 125478:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.626536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2498.666630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2498.685986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2498.716555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2498.771592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2498.854407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2499.579361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 2499.802252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2499.835538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2499.864396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2499.893836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2499.922623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2499.951788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2499.991988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2500.031764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2500.092947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2500.126503] LNet: 125597:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2500.144477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2500.182830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2500.210861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2500.244492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2500.275991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2500.306915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2501.010749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2501.210037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2501.225817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2501.261969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2501.297886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2501.338795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2501.378530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2501.424517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2501.466186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2501.537975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2501.614924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2501.668427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2501.700393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2501.737531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2501.780864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2501.817584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2507.831543] LNet: 125798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2507.842832] LNetError: 125798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2507.851181] LNet: Removed LNI 192.168.201.22@tcp [ 2507.862805] LNet: Removed LNI 192.168.201.22@tcp1 [ 2508.272944] Key type .llcrypt unregistered [ 2508.279227] Key type ._llcrypt unregistered [ 2526.527886] Key type ._llcrypt registered [ 2526.531144] Key type .llcrypt registered [ 2528.015168] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 20:55:25 (1789520125) [ 2528.443600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2528.473273] alg: No test for adler32 (adler32-zlib) [ 2529.449619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2529.500804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2529.524214] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2530.549418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2530.576399] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 2531.451793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 2531.546847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2532.639202] LNet: Removed LNI 192.168.201.22@tcp [ 2538.837362] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 20:55:36 (1789520136) [ 2540.185869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2541.829271] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:55:39 (1789520139) [ 2548.166540] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:55:46 (1789520146) [ 2555.113860] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:55:53 (1789520153) [ 2563.011605] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:56:00 (1789520160) [ 2570.586675] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:56:08 (1789520168) [ 2578.038812] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:56:15 (1789520175) [ 2585.129433] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:56:23 (1789520183) [ 2591.898968] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:56:29 (1789520189) [ 2600.254549] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:56:38 (1789520198) [ 2607.743982] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:56:45 (1789520205) [ 2614.536642] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:56:52 (1789520212) [ 2621.469776] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:56:59 (1789520219) [ 2629.181827] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:57:07 (1789520227) [ 2636.992497] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:57:14 (1789520234) [ 2644.285955] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:57:22 (1789520242) [ 2652.215963] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:57:30 (1789520250) [ 2659.499639] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:57:37 (1789520257) [ 2667.920710] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:57:45 (1789520265) [ 2668.342654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2668.375994] LNet: 137489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2668.387486] LNetError: 137489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2668.405154] LNet: Removed LNI 192.168.201.22@tcp2 [ 2668.433889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2668.604706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2668.619383] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2670.316359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2670.356826] LNet: 137631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2670.361897] LNetError: 137631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2670.389889] LNet: Removed LNI 192.168.201.22@tcp [ 2670.424044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2670.525760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2670.564474] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2677.649232] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:57:55 (1789520275) [ 2685.828576] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:58:03 (1789520283) [ 2694.281346] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:58:12 (1789520292) [ 2703.046927] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:58:21 (1789520301) [ 2711.586900] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:58:29 (1789520309) [ 2721.045911] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:58:38 (1789520318) [ 2728.708211] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:58:46 (1789520326) [ 2736.826215] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:58:54 (1789520334) [ 2744.716563] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:59:02 (1789520342) [ 2754.224511] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:59:12 (1789520352) [ 2763.050488] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:59:20 (1789520360) [ 2770.978211] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:59:29 (1789520369) [ 2778.633914] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2542 sec ========== 20:59:36 (1789520376) [ 2778.966123] LNet: 145101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2778.977700] LNetError: 145101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2778.991759] LNet: Removed LNI 192.168.201.22@tcp [ 2779.589682] Key type .llcrypt unregistered [ 2779.592436] Key type ._llcrypt unregistered [ 2792.449706] Key type ._llcrypt registered [ 2792.453723] Key type .llcrypt registered [ 2793.054505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2793.067543] alg: No test for adler32 (adler32-zlib) [ 2794.178632] Lustre: Lustre: Build Version: 2.17.58_39_g97904ba [ 2794.421305] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 2796.143360] Key type lgssc registered [ 2797.391711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2871.014233] Lustre: Mounted lustre-client - version 2.17.58_39_g97904ba [ 2877.614414] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2887.212680] Lustre: Unmounted lustre-client [ 2907.242764] Key type lgssc unregistered [ 2907.497282] LNet: 148065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.505605] LNetError: 148065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.522895] LNet: Removed LNI 192.168.201.22@tcp [ 2907.971282] Key type .llcrypt unregistered [ 2907.973749] Key type ._llcrypt unregistered