[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 444597050 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2852280K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003223] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011075] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021597] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026277] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031215] *** VALIDATE tmpfs *** [ 0.032510] *** VALIDATE proc *** [ 0.033253] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036064] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040034] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.043842] debug: unmapping init [mem 0xffffffff93259000-0xffffffff93260fff] [ 0.045266] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046598] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047036] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050028] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054380] rcu: Hierarchical SRCU implementation. [ 0.056531] smp: Bringing up secondary CPUs ... [ 0.057719] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.061534] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132967] node 0 deferred pages initialised in 67ms [ 0.136164] devtmpfs: initialized [ 0.137379] x86/mm: Memory block size: 128MB [ 0.139872] gcov: version magic: 0x41383552 [ 0.141377] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145071] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146296] pinctrl core: initialized pinctrl subsystem [ 0.147220] [ 0.147985] ************************************************************* [ 0.148016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149014] ** ** [ 0.150013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151016] ** ** [ 0.152018] ** This means that this kernel is built to expose internal ** [ 0.153013] ** IOMMU data structures, which may compromise security on ** [ 0.154013] ** your system. ** [ 0.155013] ** ** [ 0.156014] ** If you see this message and you are not debugging the ** [ 0.157015] ** kernel, report this immediately to your vendor! ** [ 0.158012] ** ** [ 0.159079] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160019] ************************************************************* [ 0.161703] NET: Registered protocol family 16 [ 0.162470] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165435] cpuidle: using governor menu [ 0.168864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171651] PCI: Using configuration type 1 for base access [ 0.174132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.185127] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191070] cryptd: max_cpu_qlen set to 1000 [ 0.193276] ACPI: Added _OSI(Module Device) [ 0.195015] ACPI: Added _OSI(Processor Device) [ 0.196009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.197013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202614] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.210363] ACPI: Interpreter enabled [ 0.211062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.212015] ACPI: Using IOAPIC for interrupt routing [ 0.213094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223619] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238084] acpiphp: Slot [2] registered [ 0.241286] acpiphp: Slot [5] registered [ 0.243336] acpiphp: Slot [6] registered [ 0.246154] acpiphp: Slot [3] registered [ 0.248273] acpiphp: Slot [4] registered [ 0.250110] acpiphp: Slot [7] registered [ 0.252112] acpiphp: Slot [8] registered [ 0.254113] acpiphp: Slot [9] registered [ 0.256171] acpiphp: Slot [10] registered [ 0.258132] acpiphp: Slot [11] registered [ 0.259113] acpiphp: Slot [12] registered [ 0.261178] acpiphp: Slot [13] registered [ 0.263104] acpiphp: Slot [14] registered [ 0.265104] acpiphp: Slot [15] registered [ 0.267095] acpiphp: Slot [16] registered [ 0.269111] acpiphp: Slot [17] registered [ 0.271115] acpiphp: Slot [18] registered [ 0.272187] acpiphp: Slot [19] registered [ 0.274101] acpiphp: Slot [20] registered [ 0.275098] acpiphp: Slot [21] registered [ 0.277106] acpiphp: Slot [22] registered [ 0.278107] acpiphp: Slot [23] registered [ 0.280102] acpiphp: Slot [24] registered [ 0.282104] acpiphp: Slot [25] registered [ 0.283131] acpiphp: Slot [26] registered [ 0.285113] acpiphp: Slot [27] registered [ 0.287157] acpiphp: Slot [28] registered [ 0.288135] acpiphp: Slot [29] registered [ 0.290109] acpiphp: Slot [30] registered [ 0.291166] acpiphp: Slot [31] registered [ 0.293066] PCI host bridge to bus 0000:00 [ 0.295020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.311192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315149] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318514] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.332491] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.343547] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.346911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.350045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.353934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.358015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.368637] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.374026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380142] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422817] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.430014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.437018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469327] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.473863] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.477505] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.480439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.483317] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.488124] iommu: Default domain type: Passthrough [ 0.489491] SCSI subsystem initialized [ 0.491173] ACPI: bus type USB registered [ 0.493112] usbcore: registered new interface driver usbfs [ 0.495100] usbcore: registered new interface driver hub [ 0.497155] usbcore: registered new device driver usb [ 0.500200] pps_core: LinuxPPS API ver. 1 registered [ 0.502012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.506180] PTP clock support registered [ 0.509069] EDAC MC: Ver: 3.0.0 [ 0.511128] PCI: Using ACPI for IRQ routing [ 0.514500] NetLabel: Initializing [ 0.515008] NetLabel: domain hash size = 128 [ 0.516009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.518070] NetLabel: unlabeled traffic allowed by default [ 0.520129] vgaarb: loaded [ 0.521310] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.523013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.529789] clocksource: Switched to clocksource kvm-clock [ 0.640451] VFS: Disk quotas dquot_6.6.0 [ 0.641996] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.644109] *** VALIDATE ramfs *** [ 0.645108] *** VALIDATE hugetlbfs *** [ 0.646387] pnp: PnP ACPI init [ 0.648415] pnp: PnP ACPI: found 6 devices [ 0.662582] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.665302] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.667086] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.668747] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.670593] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.672522] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.674976] NET: Registered protocol family 2 [ 0.678089] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.683853] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.687805] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.692602] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.695633] TCP: Hash tables configured (established 65536 bind 65536) [ 0.698190] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.700874] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.703302] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705720] NET: Registered protocol family 1 [ 0.708217] RPC: Registered named UNIX socket transport module. [ 0.710870] RPC: Registered udp transport module. [ 0.713222] RPC: Registered tcp transport module. [ 0.715356] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.718332] NET: Registered protocol family 44 [ 0.720017] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.721879] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.724028] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.726130] PCI: CLS 0 bytes, default 64 [ 0.728527] Unpacking initramfs... [ 2.121867] debug: unmapping init [mem 0xffff91f27cc64000-0xffff91f27ffcffff] [ 2.125789] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.129283] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.131674] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.637193] Initialise system trusted keyrings [ 2.639097] Key type blacklist registered [ 2.640776] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.650759] zbud: loaded [ 2.654170] *** VALIDATE nfs *** [ 2.655644] *** VALIDATE nfs4 *** [ 2.657655] pstore: using deflate compression [ 2.661409] Platform Keyring initialized [ 2.774489] NET: Registered protocol family 38 [ 2.776574] Key type asymmetric registered [ 2.778203] Asymmetric key parser 'x509' registered [ 2.780419] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.783827] io scheduler mq-deadline registered [ 2.785806] io scheduler kyber registered [ 2.787590] io scheduler bfq registered [ 2.789527] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.792384] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.795490] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.798710] ACPI: Power Button [PWRF] [ 2.804777] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.812191] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.827951] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.856876] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.886487] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.892558] Non-volatile memory driver v1.3 [ 2.893996] Linux agpgart interface v0.103 [ 2.933205] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.937464] vda: detected capacity change from 0 to 75083776 [ 2.952212] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.955749] vdb: detected capacity change from 0 to 1073741824 [ 2.962454] libphy: Fixed MDIO Bus: probed [ 2.972325] usbcore: registered new interface driver usbserial_generic [ 2.975715] usbserial: USB Serial support registered for generic [ 2.978261] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.982897] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.985127] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.987746] mousedev: PS/2 mouse device common for all mice [ 2.991101] rtc_cmos 00:05: RTC can wake from S4 [ 2.993954] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.997763] rtc_cmos 00:05: registered as rtc0 [ 3.000488] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.003837] intel_pstate: CPU model not supported [ 3.006451] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.012271] hid: raw HID events driver (C) Jiri Kosina [ 3.012447] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.014407] usbcore: registered new interface driver usbhid [ 3.019316] usbhid: USB HID core driver [ 3.020595] drop_monitor: Initializing network drop monitor service [ 3.022740] Initializing XFRM netlink socket [ 3.024456] NET: Registered protocol family 10 [ 3.027541] Segment Routing with IPv6 [ 3.029118] NET: Registered protocol family 17 [ 3.031228] mpls_gso: MPLS GSO support [ 3.037155] RAS: Correctable Errors collector initialized. [ 3.039602] AVX version of gcm_enc/dec engaged. [ 3.041423] AES CTR mode by8 optimization enabled [ 3.113712] sched_clock: Marking stable (3113617762, 0)->(4077707244, -964089482) [ 3.118475] registered taskstats version 1 [ 3.120670] Loading compiled-in X.509 certificates [ 3.123671] zswap: loaded using pool lzo/zbud [ 3.148355] Key type big_key registered [ 3.161383] Key type encrypted registered [ 3.163225] ima: No TPM chip found, activating TPM-bypass! [ 3.165738] ima: Allocated hash algorithm: sha1 [ 3.167976] ima: No architecture policies found [ 3.169937] evm: Initialising EVM extended attributes: [ 3.172403] evm: security.selinux [ 3.173731] evm: security.ima [ 3.174894] evm: security.capability [ 3.176364] evm: HMAC attrs: 0x1 [ 3.178788] rtc_cmos 00:05: setting system clock to 2026-09-08 11:12:57 UTC (1788865977) [ 3.184362] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 3.187607] debug: unmapping init [mem 0xffffffff92f82000-0xffffffff93258fff] [ 3.197061] Write protecting the kernel read-only data: 28672k [ 3.200657] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 3.203967] debug: unmapping init [mem 0xffffffff91f14000-0xffffffff91ffffff] [ 3.236429] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.246725] systemd[1]: Detected virtualization kvm. [ 3.248716] systemd[1]: Detected architecture x86-64. [ 3.250268] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.273694] systemd[1]: No hostname configured. [ 3.275958] systemd[1]: Set hostname to . [ 3.278600] random: systemd: uninitialized urandom read (16 bytes read) [ 3.281698] systemd[1]: Initializing machine ID from random generator. [ 3.421202] random: systemd: uninitialized urandom read (16 bytes read) [ 3.423641] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.428973] random: systemd: uninitialized urandom read (16 bytes read) [ 3.431847] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.438584] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.047388] device-mapper: uevent: version 1.0.3 [ 4.049772] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.800712] virtio_net virtio0 ens2: renamed from eth0 [ 4.845342] scsi host0: ata_piix [ 4.863465] scsi host1: ata_piix [ 4.865377] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.868498] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.565708] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.680990] random: crng init done [ 9.682659] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.095836] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.317334] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.579328] SELinux: Disabled at runtime. [ 11.640723] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.649807] systemd[1]: Detected virtualization kvm. [ 11.651679] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.178063] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.181479] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.188286] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.194376] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.198599] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.206503] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.217230] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ 12.320673] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.686969] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.040640] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.095515] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.217235] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.229614] EDAC sbridge: Ver: 1.1.2 [ 14.394966] Key type dns_resolver registered [ 14.697968] NFS: Registering the id_resolver key type [ 14.700368] Key type id_resolver registered [ 14.701787] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-client login: [ 71.396072] libcfs: loading out-of-tree module taints kernel. [ 71.461127] Key type ._llcrypt registered [ 71.471557] Key type .llcrypt registered [ 71.895333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.942726] alg: No test for adler32 (adler32-zlib) [ 73.445870] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 74.800250] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 76.560194] Key type lgssc registered [ 78.586454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 153.010016] hrtimer: interrupt took 5562552 ns [ 209.869239] Lustre: Mounted lustre-client - version 2.17.57_104_g1a15c94 [ 214.646601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 228.757302] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing check_logdir /tmp/testlogs/ [ 233.557363] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing yml_node [ 235.488203] Lustre: lustre-OST0000-osc-ffff91f2d9287800: disconnect after 23s idle [ 237.956506] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 240.426684] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 242.959495] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 244.886854] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 07:16:57 EDT 2026 [ 259.990314] Lustre: Unmounted lustre-client [ 281.806980] Key type lgssc unregistered [ 282.080567] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.094212] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.116463] LNet: Removed LNI 192.168.203.16@tcp [ 282.950194] Key type .llcrypt unregistered [ 282.953264] Key type ._llcrypt unregistered [ 304.923909] Key type ._llcrypt registered [ 304.931135] Key type .llcrypt registered [ 305.198085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 305.215069] alg: No test for adler32 (adler32-zlib) [ 306.160691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 306.240743] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 306.268585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 310.459671] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 310.605969] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.625339] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.640760] LNet: Removed LNI 192.168.203.16@tcp [ 311.218680] Key type .llcrypt unregistered [ 311.222322] Key type ._llcrypt unregistered [ 316.837253] Key type ._llcrypt registered [ 316.844114] Key type .llcrypt registered [ 318.317583] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:18:10 (1788866290) [ 318.609417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.622125] alg: No test for adler32 (adler32-zlib) [ 319.549301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.638563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.665405] LNet: 9702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.681642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.742424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.757961] LNet: 9709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 325.431927] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:18:18 (1788866298) [ 325.531503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.568967] LNet: 10280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.598350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.705710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 331.372867] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:18:24 (1788866304) [ 331.495602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.550479] LNet: 10871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.586178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.661573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 337.363536] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:18:30 (1788866310) [ 337.463175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.489545] LNet: 11456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.532904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.580920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 343.423869] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:18:36 (1788866316) [ 343.571590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.585366] LNet: 12040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.609985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.715893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 344.448160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 352.650575] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:18:45 (1788866325) [ 352.818513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.837460] LNet: 12632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.901941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.980826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 353.342853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 353.883297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 360.994869] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:18:53 (1788866333) [ 361.606800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.681704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.824367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 368.757770] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:19:01 (1788866341) [ 368.872368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.908322] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.928093] LNet: 13808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 368.969759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.132988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 369.192872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.694986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 369.745437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 370.483604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 370.537527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 371.081191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 371.103242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 371.713574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 371.774333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.320361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 372.378966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 372.965442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 373.001302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 373.398796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 373.442101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.055503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 374.126292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 374.651507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 374.712268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.252689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 375.285324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 375.764714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 375.804963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.255583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 376.305511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 383.930310] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:19:16 (1788866356) [ 384.055476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.119723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.284230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 384.380326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 390.679719] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:19:23 (1788866363) [ 390.812590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.865803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.958402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 390.995587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 397.430519] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:19:30 (1788866370) [ 397.518572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.582595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.718973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 397.797621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 404.318766] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:19:37 (1788866377) [ 404.795748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.845367] LNet: 16255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.852261] LNet: 16255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 404.903492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.018836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 405.083300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 411.307504] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:19:44 (1788866384) [ 411.494539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.552295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.662549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.700992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 416.766565] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:19:50 (1788866390) [ 416.846234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.869423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.945718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 416.979621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.894437] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:19:55 (1788866395) [ 422.094667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.124816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.199739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.457676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 428.326295] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:20:01 (1788866401) [ 428.421119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.464484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.675895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.865899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.262980] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:20:08 (1788866408) [ 435.399622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.432485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.601396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.861502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.042329] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:20:16 (1788866416) [ 443.363695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.409517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.106167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.484793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 451.198576] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:20:24 (1788866424) [ 451.611941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.671797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.757224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.805189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.377557] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:20:30 (1788866430) [ 457.512292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.547544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.602693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.639378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.380161] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:20:36 (1788866436) [ 463.547596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.586448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.646854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 470.101444] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:20:43 (1788866443) [ 470.315693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.365990] LNet: 22278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.374057] LNet: 22278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 470.407767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.489914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 476.636131] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:20:49 (1788866449) [ 476.737530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.770601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.869231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 477.343821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 483.053123] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:20:56 (1788866456) [ 483.333213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.411054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.490357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 483.529292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 489.108637] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:21:02 (1788866462) [ 489.188114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.214884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.303988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 489.328733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 494.611125] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:21:07 (1788866467) [ 494.871946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.917236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.025722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 495.070366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 502.035035] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:21:14 (1788866474) [ 502.294394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.379975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.456418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 502.520103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.357849] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:21:21 (1788866481) [ 508.466183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.524914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.568501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 508.620921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 508.641054] LNetError: 25809:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 508.668442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 508.697257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 514.155760] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:21:27 (1788866487) [ 514.351158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.405962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.483274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OCwipC0fZl [ 514.546597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OCwipC0fZl [ 514.584579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.OCwipC0fZl [ 520.043957] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:21:33 (1788866493) [ 520.236330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.265589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.307863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 520.346215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 520.398854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 526.247458] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:21:39 (1788866499) [ 526.472820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.506599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.564220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.590993] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 528.213739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 529.370772] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 531.307281] Key type lgssc registered [ 532.655387] Lustre: Echo OBD driver; http://www.lustre.org/ [ 536.799982] LNetError: 27570:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.16@tcp99 has no available nets [ 539.561384] Key type lgssc unregistered [ 540.046478] LNetError: 28676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.068185] LNet: Removed LNI 192.168.203.16@tcp [ 540.853346] Key type .llcrypt unregistered [ 540.860197] Key type ._llcrypt unregistered [ 549.108260] Key type ._llcrypt registered [ 549.113389] Key type .llcrypt registered [ 550.806786] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:22:03 (1788866523) [ 551.518886] Key type .llcrypt unregistered [ 551.525129] Key type ._llcrypt unregistered [ 558.735409] Key type ._llcrypt registered [ 558.739478] Key type .llcrypt registered [ 558.992274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 559.027317] alg: No test for adler32 (adler32-zlib) [ 560.000176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.112471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 560.158482] LNet: 29600:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 560.172082] LNet: 29600:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 560.183584] LNet: 29600:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 560.198659] LNet: Message forwarding will be enabled in 2s [ 563.296117] LNet: Message forwarding enabled [ 563.677516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 563.713737] LNet: Message forwarding will stop in 2s to 6s [ 567.395925] LNet: Message forwarding disabled [ 568.181579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 568.198803] LNet: 29698:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 568.206371] LNet: 29698:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 568.214526] LNet: 29698:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 568.226084] LNet: Message forwarding will be enabled in 2s [ 568.382220] LNet: 29711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.051581] Key type .llcrypt unregistered [ 569.054526] Key type ._llcrypt unregistered [ 579.539448] Key type ._llcrypt registered [ 579.541614] Key type .llcrypt registered [ 580.668547] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:22:33 (1788866553) [ 581.117404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 581.146322] alg: No test for adler32 (adler32-zlib) [ 582.066809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.147644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 588.044816] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:22:40 (1788866560) [ 588.321163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.369675] LNet: 31111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.401360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.467750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 588.511341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 588.543639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 588.577261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 588.616339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 588.647386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 588.682483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 588.740945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 588.804369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 588.867384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 588.941492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 588.985872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 589.026752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 589.073346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 589.120750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 589.153491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 589.200864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 589.255232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 589.307537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 589.331760] LNetError: 31154:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 596.312030] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:22:49 (1788866569) [ 596.532731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.571946] LNet: 31726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.602531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.701285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.563908] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:22:55 (1788866575) [ 603.973445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg316-server [ 605.631676] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:22:58 (1788866578) [ 605.715283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.745338] LNet: 32649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.771684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.823336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.857554] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 606.677108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[17-21]@tcp [ 606.692992] LNet: 32789:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 606.740296] LNetError: 32656:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 606.764306] LNetError: 32656:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.17@tcp has gone from up to down [ 614.025040] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:23:06 (1788866586) [ 616.242561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg316-server [ 618.186104] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:23:10 (1788866590) [ 618.448382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.489501] LNet: 33725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.512376] LNetError: 33725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.561037] LNet: Removed LNI 192.168.203.16@tcp [ 619.602972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.695414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 619.734796] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 620.650616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 620.677274] LNet: 33842:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 620.698594] LNet: 33842:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 620.726649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 620.735488] LNetError: 33729:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.19@tcp is being used as a gateway but routing feature is not turned on [ 620.751115] LNetError: 33729:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 620.761417] LNetError: 33729:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.19@tcp has gone from up to down [ 620.770156] LNetError: 33729:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 628.836565] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:23:21 (1788866601) [ 628.976786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.017806] LNet: 34429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.029671] LNetError: 34429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.054714] LNet: Removed LNI 192.168.203.16@tcp [ 630.088047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.202837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 630.283947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.325105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.396451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.463778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.500456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.572606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 630.652490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.691055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.761119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.852908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.892364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.962162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 631.034941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.060797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.113811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.182948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.217656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.263475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 631.340034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.388456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.495421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.560661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.604356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.671446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.748523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.774993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.821351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.873398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.902238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.938584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 631.983416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.014673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.064731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.160396] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:23:31 (1788866611) [ 639.973938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg316-server [ 641.413990] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:23:34 (1788866614) [ 643.299792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg316-server [ 644.787430] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:23:37 (1788866617) [ 645.061889] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.076301] LNet: 35836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 645.716476] Key type .llcrypt unregistered [ 645.722312] Key type ._llcrypt unregistered [ 651.793285] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 651.977661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 651.989838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 651.995402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 654.433431] Key type ._llcrypt registered [ 654.435632] Key type .llcrypt registered [ 654.590983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.619989] alg: No test for adler32 (adler32-zlib) [ 655.666968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.725813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.743291] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 656.454520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 656.474824] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 658.209376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 658.229684] LNet: Removed LNI 10.1.2.3@tcp [ 658.359572] LNet: 36558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.370575] LNetError: 36558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.385897] LNet: Removed LNI 192.168.203.16@tcp [ 659.154558] Key type .llcrypt unregistered [ 659.156950] Key type ._llcrypt unregistered [ 672.684418] Key type ._llcrypt registered [ 672.691294] Key type .llcrypt registered [ 674.282473] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:24:06 (1788866646) [ 674.784449] Key type .llcrypt unregistered [ 674.788353] Key type ._llcrypt unregistered [ 681.078990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 681.307184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 681.317986] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 681.328961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 683.946540] Key type ._llcrypt registered [ 683.950759] Key type .llcrypt registered [ 684.177185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 684.197395] alg: No test for adler32 (adler32-zlib) [ 685.174693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.241824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.272700] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 685.990561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 686.028364] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 688.961913] LNet: 38049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.981790] LNetError: 38049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 688.993443] LNet: Removed LNI 10.1.2.3@tcp [ 689.012261] LNet: Removed LNI 192.168.203.16@tcp [ 689.659460] Key type .llcrypt unregistered [ 689.664700] Key type ._llcrypt unregistered [ 698.162369] Key type ._llcrypt registered [ 698.164952] Key type .llcrypt registered [ 699.164967] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:24:32 (1788866672) [ 699.538550] Key type .llcrypt unregistered [ 699.545455] Key type ._llcrypt unregistered [ 703.270639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 703.393765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 703.403352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 703.412391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.848454] Key type ._llcrypt registered [ 705.850682] Key type .llcrypt registered [ 706.020654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.032346] alg: No test for adler32 (adler32-zlib) [ 706.936994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.008226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 707.051529] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 708.432648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.453145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.157623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 710.183579] LNet: Removed LNI 10.1.2.3@tcp [ 710.195501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 710.219975] LNet: Removed LNI 10.1.2.31@tcp [ 710.242251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 710.280933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.998938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 711.029239] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 711.956462] LNet: 39964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.963459] LNetError: 39964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 711.971745] LNet: Removed LNI 10.1.2.31@tcp [ 711.985245] LNet: Removed LNI 10.1.2.3@tcp [ 712.592338] Key type .llcrypt unregistered [ 712.598936] Key type ._llcrypt unregistered [ 722.031345] Key type ._llcrypt registered [ 722.033753] Key type .llcrypt registered [ 723.006978] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:24:56 (1788866696) [ 723.349891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 723.361708] alg: No test for adler32 (adler32-zlib) [ 724.214715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.271823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.296579] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 724.881406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.901560] LNet: 40914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.907436] LNetError: 40914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.927760] LNet: Removed LNI 192.168.203.16@tcp [ 724.946302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.983897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.16@tcp [ 725.000774] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 725.487251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.16@tcp [ 725.506793] LNet: Removed LNI 192.168.203.16@tcp [ 725.558720] LNet: 41049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.137577] Key type .llcrypt unregistered [ 726.143397] Key type ._llcrypt unregistered [ 735.659348] Key type ._llcrypt registered [ 735.662632] Key type .llcrypt registered [ 736.983223] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:25:09 (1788866709) [ 739.480779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg316-server [ 741.411619] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:25:14 (1788866714) [ 741.997736] Key type .llcrypt unregistered [ 742.003917] Key type ._llcrypt unregistered [ 747.603541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.712758] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.717398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.726832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.309049] Key type ._llcrypt registered [ 750.311396] Key type .llcrypt registered [ 750.602521] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.620867] alg: No test for adler32 (adler32-zlib) [ 751.585615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.663650] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 752.465653] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 755.042955] LNet: 42881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.058293] LNetError: 42881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.070324] LNet: Removed LNI 192.168.203.16@tcp1 [ 755.094528] LNet: Removed LNI 10.1.2.3@tcp2 [ 755.920402] Key type .llcrypt unregistered [ 755.923148] Key type ._llcrypt unregistered [ 767.632251] Key type ._llcrypt registered [ 767.635408] Key type .llcrypt registered [ 769.182266] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:25:41 (1788866741) [ 769.707598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 769.735430] alg: No test for adler32 (adler32-zlib) [ 770.692295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.738919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 770.759220] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 771.749086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.779547] LNet: 43849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.785482] LNetError: 43849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.804733] LNet: Removed LNI 192.168.203.16@tcp [ 771.816983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.867253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 771.892786] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 778.222156] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:25:51 (1788866751) [ 778.513370] LNet: 44440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.519330] LNetError: 44440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.535990] LNet: Removed LNI 192.168.203.16@tcp [ 779.334784] Key type .llcrypt unregistered [ 779.337322] Key type ._llcrypt unregistered [ 785.356868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.609417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.634641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.645141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.151740] Key type ._llcrypt registered [ 788.155830] Key type .llcrypt registered [ 788.306686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.317878] alg: No test for adler32 (adler32-zlib) [ 789.203990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.254858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.275601] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 789.809941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 789.829450] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.764688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.802439] LNet: 45180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.816452] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 791.821991] LNet: Removed LNI 10.1.2.3@tcp [ 791.835503] LNet: Removed LNI 192.168.203.16@tcp [ 791.850639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.903495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 791.953417] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 791.965306] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.312419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.327810] LNet: 45207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.333709] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.338862] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.343985] LNet: Removed LNI 10.1.2.3@tcp [ 792.352365] LNet: Removed LNI 192.168.203.16@tcp [ 792.368417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.421451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 792.453455] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 792.467340] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.688349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 792.714582] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 793.569430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.591893] LNet: 45369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.600684] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 793.607057] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 793.613321] LNet: Removed LNI 10.1.2.3@tcp [ 793.617115] LNet: Removed LNI 192.168.203.16@tcp [ 793.639775] LNet: Removed LNI 192.168.203.16@tcp2 [ 793.659881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.747452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 793.767743] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 793.778607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 793.784562] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 799.209227] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:26:12 (1788866772) [ 799.365119] LNet: 45960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.373664] LNetError: 45960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.380594] LNetError: 45960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 799.395429] LNet: Removed LNI 10.1.2.3@tcp [ 799.398121] LNet: Removed LNI 192.168.203.16@tcp [ 799.423370] LNet: Removed LNI 192.168.203.16@tcp2 [ 799.953507] Key type .llcrypt unregistered [ 799.958073] Key type ._llcrypt unregistered [ 804.765215] Key type ._llcrypt registered [ 804.771882] Key type .llcrypt registered [ 804.961403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.976863] alg: No test for adler32 (adler32-zlib) [ 805.857596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.903680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.925104] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 806.644622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.667163] LNet: 46359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.675828] LNetError: 46359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.686905] LNet: Removed LNI 192.168.203.16@tcp [ 806.705772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.758765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 806.795694] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 807.282938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.304603] LNet: 46389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.313257] LNetError: 46389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.336101] LNet: Removed LNI 192.168.203.16@tcp [ 807.359241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.408448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 807.434114] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 807.802083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.823555] LNet: Removed LNI 192.168.203.16@tcp [ 807.837831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.884500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 807.903131] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 808.289184] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 808.445590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.451817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 808.458692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 810.712496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.746912] LNet: 46497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.751071] LNet: 46497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 810.756195] LNetError: 46497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.760785] LNetError: 46497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 810.775738] LNet: Removed LNI 192.168.203.16@tcp [ 810.794970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.852648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 810.877167] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 815.767383] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:26:29 (1788866789) [ 815.931527] LNet: 47093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.939644] LNetError: 47093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.953228] LNet: Removed LNI 192.168.203.16@tcp [ 816.451507] Key type .llcrypt unregistered [ 816.456961] Key type ._llcrypt unregistered [ 821.327487] Key type ._llcrypt registered [ 821.330768] Key type .llcrypt registered [ 821.552631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.577311] alg: No test for adler32 (adler32-zlib) [ 822.471416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.526217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.551412] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 823.297038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 823.317684] LNet: 47490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.323330] LNetError: 47490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.340956] LNet: Removed LNI 192.168.203.16@tcp [ 823.361709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.436767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 823.460723] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 829.051924] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:26:42 (1788866802) [ 829.110419] LNet: 48073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.116145] LNetError: 48073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.131830] LNet: Removed LNI 192.168.203.16@tcp [ 829.616751] Key type .llcrypt unregistered [ 829.619219] Key type ._llcrypt unregistered [ 834.182526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 834.268735] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.276453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 834.283021] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 836.891603] Key type ._llcrypt registered [ 836.894460] Key type .llcrypt registered [ 837.053769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.069337] alg: No test for adler32 (adler32-zlib) [ 837.940437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.988675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.008034] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 838.617641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.632248] LNet: 48530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.640579] LNetError: 48530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.668971] LNet: Removed LNI 192.168.203.16@tcp [ 838.685386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.741165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 838.765356] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 843.994970] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:26:57 (1788866817) [ 844.176564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.206352] LNet: 49128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.212932] LNetError: 49128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.239833] LNet: Removed LNI 192.168.203.16@tcp [ 844.279731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.355361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 850.006766] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:27:02 (1788866822) [ 850.081540] LNet: 49704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.786703] Key type .llcrypt unregistered [ 850.793670] Key type ._llcrypt unregistered [ 856.259641] Key type ._llcrypt registered [ 856.261800] Key type .llcrypt registered [ 856.434362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.465534] alg: No test for adler32 (adler32-zlib) [ 857.348653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.409469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.440854] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 858.222301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.241191] LNet: 50105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.249435] LNetError: 50105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.266243] LNet: Removed LNI 192.168.203.16@tcp [ 858.278224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.355780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 858.376593] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 865.300160] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:27:18 (1788866838) [ 865.371463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.408152] LNet: 50693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.416243] LNetError: 50693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.446373] LNet: Removed LNI 192.168.203.16@tcp [ 865.489556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.541525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 871.331295] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:27:24 (1788866844) [ 872.712291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 874.151325] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:27:27 (1788866847) [ 875.938518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 877.630521] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:27:30 (1788866850) [ 879.216798] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 881.156313] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:27:33 (1788866853) [ 882.399728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 884.119119] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:27:37 (1788866857) [ 885.619744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 887.338356] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:27:40 (1788866860) [ 887.565499] LNet: 52750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.351648] Key type .llcrypt unregistered [ 888.355376] Key type ._llcrypt unregistered [ 895.122499] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.306637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.312226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.319798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.767659] Key type ._llcrypt registered [ 897.769330] Key type .llcrypt registered [ 897.931452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.963191] alg: No test for adler32 (adler32-zlib) [ 898.916591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.996089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 899.020418] LNet: Added LNI 192.168.203.16@tcp [8/512/0/180] [ 899.025337] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 899.029236] LNet: 53069:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 899.131218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.148687] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.154674] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.160891] LNet: Removed LNI 10.1.2.3@tcp [ 899.164264] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.179382] LNet: Removed LNI 192.168.203.16@tcp [ 899.197495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.261529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 899.296843] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 899.362327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.386296] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.394694] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.420507] LNet: Removed LNI 192.168.203.16@tcp [ 899.450777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.530297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 899.563077] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 899.569909] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.573330] LNet: 53120:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 899.654937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.673680] LNet: 53138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.678697] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 899.684700] LNet: Removed LNI 10.1.2.3@tcp [ 899.687950] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.705941] LNet: Removed LNI 192.168.203.16@tcp [ 899.722882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.772310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 899.797107] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 899.803712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 899.809949] LNet: 53146:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 900.025918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.040042] LNet: 53172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.045367] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.050203] LNet: Removed LNI 10.1.2.3@tcp [ 900.055940] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.068204] LNet: Removed LNI 192.168.203.16@tcp [ 900.087048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.145043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 900.168251] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 900.171883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 900.246511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.259712] LNet: 53200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.264893] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 900.270139] LNet: Removed LNI 10.1.2.3@tcp [ 900.272292] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.284916] LNet: Removed LNI 192.168.203.16@tcp [ 900.297169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.348308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 900.364973] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 900.371164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 900.374434] LNet: 53208:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 900.479499] LNet: Removed LNI 10.1.2.3@tcp [ 900.491104] LNet: Removed LNI 192.168.203.16@tcp [ 901.067743] Key type .llcrypt unregistered [ 901.072962] Key type ._llcrypt unregistered [ 912.274871] Key type ._llcrypt registered [ 912.280501] Key type .llcrypt registered [ 913.391872] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:28:06 (1788866886) [ 914.430625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 915.955491] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:28:09 (1788866889) [ 916.361745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 916.381903] alg: No test for adler32 (adler32-zlib) [ 917.275345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.325393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.341508] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 917.890130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.16 [ 917.905232] LNet: 54471:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 917.912204] LNet: Removed LNI 192.168.203.16@tcp [ 922.986793] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:28:16 (1788866896) [ 923.131292] LNet: 55036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.147771] LNetError: 55036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.167164] LNet: Removed LNI 192.168.203.16@tcp [ 923.738359] Key type .llcrypt unregistered [ 923.741479] Key type ._llcrypt unregistered [ 929.830422] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.084080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.584157] Key type ._llcrypt registered [ 931.590947] Key type .llcrypt registered [ 931.744126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.753277] alg: No test for adler32 (adler32-zlib) [ 933.020778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.459938] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:28:31 (1788866911) [ 938.589623] LNet: 55937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.608761] LNetError: 55937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.645950] LNet: Removed LNI 10.1.2.3@tcp [ 939.198402] Key type .llcrypt unregistered [ 939.200355] Key type ._llcrypt unregistered [ 944.708992] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.918286] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.925670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.929967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.465703] Key type ._llcrypt registered [ 947.469747] Key type .llcrypt registered [ 947.618943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.628280] alg: No test for adler32 (adler32-zlib) [ 948.542091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.142490] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:28:47 (1788866927) [ 955.601282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 957.310440] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:28:50 (1788866930) [ 957.430545] LNet: 57119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.452383] LNetError: 57119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.485696] LNet: Removed LNI 10.1.2.3@tcp [ 958.228634] Key type .llcrypt unregistered [ 958.230860] Key type ._llcrypt unregistered [ 964.167903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 964.387601] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 964.394635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 964.402623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.080227] Key type ._llcrypt registered [ 967.082429] Key type .llcrypt registered [ 967.243836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.253675] alg: No test for adler32 (adler32-zlib) [ 968.175366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.225257] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 976.363071] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:29:09 (1788866949) [ 977.026390] LNet: 58351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.033675] LNetError: 58351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.055241] LNet: Removed LNI 10.1.2.3@tcp [ 977.642875] Key type .llcrypt unregistered [ 977.645135] Key type ._llcrypt unregistered [ 983.075056] Key type ._llcrypt registered [ 983.077194] Key type .llcrypt registered [ 983.250375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.276371] alg: No test for adler32 (adler32-zlib) [ 984.211652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 984.256449] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 991.694376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 992.388552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 992.426635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 992.482571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 992.500895] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 992.743797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.774242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.802368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.839939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.872401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.902321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.931501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.956893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.973504] LNet: 58786:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.991624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.035455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.075636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.106177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.166869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.206471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.260340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 993.824877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 993.865502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 993.897595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 993.924155] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 994.159811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.208481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 994.247042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.281494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.352863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.398373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.427321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.461068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.493493] LNet: 58895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.525580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.530590] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 994.569209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.614882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.653622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.678979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.713727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.764859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 995.253505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 995.311717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 995.358234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 995.381617] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 995.393028] LNet: Skipped 1 previous similar message [ 995.597428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.626915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 995.665939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.709177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.762324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.801794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 995.880691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 995.953425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 995.997550] LNet: 59004:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.016186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.051766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.117355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.179656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.228567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.278142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.321533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 996.917145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 996.974674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 997.012571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 997.249688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 997.277430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 997.330410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 997.358560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 997.388208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 997.432483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 997.475073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 997.514129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 997.570826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.643793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.701512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.750234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.790678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.842226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.877204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 998.474545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 998.529558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 998.589854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 998.625932] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 998.632555] LNet: Skipped 1 previous similar message [ 998.892828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 998.922242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 998.946051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 998.973820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 999.000850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 999.032940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 999.071406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 999.102608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 999.120592] LNet: 59223:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 999.131214] LNet: 59223:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 999.153201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 999.182490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 999.211663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 999.239639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 999.265539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 999.295403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 999.333996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 999.893620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 999.934637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 999.993512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1000.239484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1000.265336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1000.287876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.307304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.343038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.375280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.408533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.444401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.490480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.550482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.592671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.647823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.676360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.730730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.766537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1001.130215] LNet: 59373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.136131] LNetError: 59373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.155905] LNet: Removed LNI 192.168.203.16@tcp [ 1001.705639] Key type .llcrypt unregistered [ 1001.711370] Key type ._llcrypt unregistered [ 1024.664429] Key type ._llcrypt registered [ 1024.668234] Key type .llcrypt registered [ 1027.259398] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:29:58 (1788866998) [ 1028.537251] Key type .llcrypt unregistered [ 1028.540568] Key type ._llcrypt unregistered [ 1037.750850] Key type ._llcrypt registered [ 1037.753552] Key type .llcrypt registered [ 1038.196385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.259966] alg: No test for adler32 (adler32-zlib) [ 1039.229924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1039.266266] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1047.433310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1059.297773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.346924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1059.378795] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1061.754865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1061.820116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1061.891936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1061.955830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1062.041256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1062.123162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1062.191373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1062.271518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1062.356565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1062.413531] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1062.438699] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1062.763802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.803445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.850309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.894922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.952847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.984500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.031491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.058230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.073462] LNet: 60839:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.089540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.132102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.171722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.217777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.275509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.341218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.411920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.105824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1064.162283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1064.213961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1064.267664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1064.370895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1064.440340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1064.496204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1064.556408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1064.603359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1064.629248] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1064.637524] LNet: Skipped 2 previous similar messages [ 1064.993794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.029867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.080521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.127362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.163959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.202577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.240711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.317323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.355404] LNet: 60961:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.389302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.466913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.539552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.612318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.658436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.697512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.710382] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1065.731414] LNet: Skipped 2 previous similar messages [ 1065.782376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.442703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1066.477821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1066.510284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1066.552872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1066.599548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1066.659449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1066.724404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1066.774189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1066.813280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1067.037825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.091790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1067.127912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.188034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.248754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.289513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.327973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.380712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.417593] LNet: 61082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1067.448304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.498877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.541285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.592985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.639505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.689379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.758522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1068.646791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1068.689355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1068.750332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1068.800220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1068.863226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1068.932871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1068.974202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1069.015079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1069.051540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1069.075061] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1069.086776] LNet: Skipped 3 previous similar messages [ 1069.377828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.416997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.452605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.493460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.538211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.605598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.651124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.703547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.737522] LNet: 61203:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.783515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.824065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.867834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.911931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.949911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.990454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.042324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1070.622561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1070.676802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1070.715734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1070.777693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1070.836836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1070.889560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1070.944851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1070.999880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1071.061356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1071.389275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.421270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.480537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.508973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.547568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.596021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.636835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.685765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.732794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.768485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.809395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.866907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.956508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.038539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.081426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1072.701912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1072.749919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1072.813829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1072.883394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1072.944919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1073.036276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1073.102698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1073.177381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1073.296954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1073.335071] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1073.348543] LNet: Skipped 5 previous similar messages [ 1073.770622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.822923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.889733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.937036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.986784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.056502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.110445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.161941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.189841] LNet: 61446:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.197119] LNet: 61446:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1074.221390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.253742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.304727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.340927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.394993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.447236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.497531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1081.938331] LNet: 61522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.961178] LNetError: 61522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.974232] LNet: Removed LNI 192.168.203.16@tcp [ 1083.063691] LNet: Removed LNI 192.168.203.16@tcp1 [ 1083.700882] Key type .llcrypt unregistered [ 1083.705992] Key type ._llcrypt unregistered [ 1102.188954] Key type ._llcrypt registered [ 1102.191149] Key type .llcrypt registered [ 1103.930551] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:31:16 (1788867076) [ 1104.954427] Key type .llcrypt unregistered [ 1104.956876] Key type ._llcrypt unregistered [ 1111.784395] Key type ._llcrypt registered [ 1111.789655] Key type .llcrypt registered [ 1111.995881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.020640] alg: No test for adler32 (adler32-zlib) [ 1112.937792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.980210] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1121.148695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1121.773808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1121.828819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1121.888919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1121.918492] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1122.267124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.298594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1122.329795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1122.362324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1122.399610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1122.428918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.481751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.549630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.580592] LNet: 62767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.611655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.655658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.696242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.728818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.766666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.808732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.867418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1123.544831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1123.626897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1123.681750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1123.718865] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1124.032979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.086352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.125215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.172328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.209816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.244081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.290787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.324900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.341891] LNet: 62876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.371268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.400773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.434755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.480433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.517814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.569734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.603526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.213790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1125.274803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1125.320751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1125.338920] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1125.603746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.656778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.683984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.730794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.781597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.823082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.864173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.900640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.930641] LNet: 62986:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.950094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.980885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.027897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.064699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.116818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.158217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.201160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.489715] LNet: 63027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.502209] LNetError: 63027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.518084] LNet: Removed LNI 192.168.203.16@tcp [ 1126.992424] Key type .llcrypt unregistered [ 1126.997081] Key type ._llcrypt unregistered [ 1141.767296] Key type ._llcrypt registered [ 1141.770173] Key type .llcrypt registered [ 1143.202569] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:31:55 (1788867115) [ 1143.986616] Key type .llcrypt unregistered [ 1143.993821] Key type ._llcrypt unregistered [ 1150.501419] Key type ._llcrypt registered [ 1150.503394] Key type .llcrypt registered [ 1150.790595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.823601] alg: No test for adler32 (adler32-zlib) [ 1151.747170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1151.817490] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1159.305220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1167.901156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.984191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1168.014796] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1169.473856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1169.508696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1169.543110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1169.573160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1169.608333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1169.658983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1169.691277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1169.742449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1169.794265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1169.826295] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1169.834034] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1170.058877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.088166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.108749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.141568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.182347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.228183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.263529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.294270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.311831] LNet: 64488:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.327607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.370147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.407228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.441747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.504307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.545551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.575792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.615431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.082939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1171.113491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1171.143995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1171.177448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1171.201460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1171.230093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1171.285689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1171.319745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1171.356667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1171.379865] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1171.385441] LNet: Skipped 2 previous similar messages [ 1171.609337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.633780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.660779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.691163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.712056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.730990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.752808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.775848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.800495] LNet: 64612:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.813850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.841697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.863862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.890567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.913619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.938070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.970278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.003280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.507614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1172.552493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1172.600491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1172.652464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1172.689763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1172.737117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1172.771290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1172.800968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1172.840563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1172.859576] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1173.091063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.131172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.172731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.221723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.254304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.299641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.334871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.382668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.417261] LNet: 64735:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.451323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.489559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.527927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.576365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.632235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.660704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.701323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.735644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.477355] LNet: 64813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.485074] LNetError: 64813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.496343] LNet: Removed LNI 192.168.203.16@tcp [ 1178.515801] LNet: Removed LNI 192.168.203.16@tcp1 [ 1178.963506] Key type .llcrypt unregistered [ 1178.966281] Key type ._llcrypt unregistered [ 1193.479160] Key type ._llcrypt registered [ 1193.484834] Key type .llcrypt registered [ 1194.844815] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:32:47 (1788867167) [ 1196.542992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1197.935316] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:32:51 (1788867171) [ 1198.981788] Key type .llcrypt unregistered [ 1198.984872] Key type ._llcrypt unregistered [ 1206.988215] Key type ._llcrypt registered [ 1206.992515] Key type .llcrypt registered [ 1207.282150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.311504] alg: No test for adler32 (adler32-zlib) [ 1208.331531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.387115] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1216.909930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1217.504391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1217.550443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1217.584056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1217.620555] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1217.632143] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1217.927536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.974851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1218.020360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1218.068493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1218.107459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1218.140310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.175482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.225181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.260899] LNet: 66350:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.298736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.361917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.397389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.439617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.479408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.505883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.798729] LNet: 66389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.812656] LNetError: 66389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.852685] LNet: Removed LNI 192.168.203.16@tcp [ 1219.390521] Key type .llcrypt unregistered [ 1219.395737] Key type ._llcrypt unregistered [ 1237.892467] Key type ._llcrypt registered [ 1237.900420] Key type .llcrypt registered [ 1238.156211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.178533] alg: No test for adler32 (adler32-zlib) [ 1239.197362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1239.239840] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1247.067362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1255.493623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.524511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1255.573587] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1256.888359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1256.932841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1256.985671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1257.027910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1257.064941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1257.106489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1257.136888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1257.167182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1257.193375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1257.207532] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1257.371267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.396184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.426989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.456481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.483851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.510702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.544587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.576794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.602194] LNet: 67263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.624296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.655699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.678705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.697328] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1257.704976] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1257.707674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.749906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.784482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.450291] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1262.780699] LNet: 67337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.798829] LNetError: 67337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.805497] LNet: Removed LNI 192.168.203.16@tcp [ 1262.833653] LNet: Removed LNI 192.168.203.16@tcp1 [ 1263.200654] Key type .llcrypt unregistered [ 1263.205569] Key type ._llcrypt unregistered [ 1278.426259] Key type ._llcrypt registered [ 1278.427831] Key type .llcrypt registered [ 1279.532824] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:34:12 (1788867252) [ 1279.785152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1279.796863] alg: No test for adler32 (adler32-zlib) [ 1280.679520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.729488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1280.751815] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1281.424732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1281.463795] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1282.258290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1282.331290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1282.367482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1282.401713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1282.462679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1282.503406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1282.542119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1282.565778] LNetError: 68434:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1282.784322] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1282.785079] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1284.832282] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1286.039744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1288.928602] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1289.555297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1289.599327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.653482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1289.674551] LNet: 68541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.679840] LNetError: 68541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.720408] LNet: Removed LNI 192.168.203.16@tcp [ 1290.732126] LNetError: 68541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.754777] LNet: Removed LNI 192.168.203.16@tcp1 [ 1290.799077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.856384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1290.893211] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1291.563411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1291.584354] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1292.435237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1292.501501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.560748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1292.618579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1292.668439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1292.712400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1292.765961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1292.813375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1292.848279] LNetError: 68814:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1292.897689] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1295.232601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1298.016636] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1298.022963] LNet: Skipped 1 previous similar message [ 1303.076829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1303.107447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1303.131830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1309.389687] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:34:42 (1788867282) [ 1309.600870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1309.629609] LNet: 69521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.635753] LNetError: 69521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.688196] LNet: Removed LNI 192.168.203.16@tcp [ 1310.718216] LNet: Removed LNI 192.168.203.16@tcp1 [ 1310.783744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.877678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1310.947207] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1311.724947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1311.755147] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1312.739611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1312.835349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1312.861828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1312.936835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1313.009237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1313.072519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1313.117975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1313.288929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1313.888819] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.16@tcp [ 1314.913055] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1314.920099] LNet: Skipped 1 previous similar message [ 1329.199699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.428504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1329.478603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.545397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 500 [ 1329.686409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.461306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.558172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.599548] LNet: 70013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.608047] LNetError: 70013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.624995] LNetError: 70013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1336.636127] LNet: Removed LNI 192.168.203.16@tcp [ 1336.649199] LNet: Removed LNI 192.168.203.16@tcp1 [ 1336.688860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.745291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.770226] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1337.773096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.828720] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1339.378450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1339.426188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.550061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1339.588486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1339.629684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1339.695437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1339.793354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1339.834028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1340.833423] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1340.850434] LNet: Skipped 3 previous similar messages [ 1341.254040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.722672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.749891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1350.784557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.306772] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:35:30 (1788867330) [ 1357.773492] LNet: 71050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.787590] LNetError: 71050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.796368] LNetError: 71050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.808776] LNet: Removed LNI 192.168.203.16@tcp [ 1357.831712] LNet: Removed LNI 192.168.203.16@tcp1 [ 1358.507763] Key type .llcrypt unregistered [ 1358.509478] Key type ._llcrypt unregistered [ 1364.606777] Key type ._llcrypt registered [ 1364.612541] Key type .llcrypt registered [ 1364.819599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.832634] alg: No test for adler32 (adler32-zlib) [ 1365.786668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1365.811493] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1373.959733] Lustre: DEBUG MARKER: Initial discovery [ 1374.034160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1376.143799] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1376.306692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1376.345586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1376.363335] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1376.391880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.452376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -l 3 [ 1376.487945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -e local_error [ 1376.537535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m PUT -l 6 [ 1377.120800] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1377.721293] Lustre: DEBUG MARKER: Force oleg316-client.virtnet to discover 192.168.203.116@tcp (in background) [ 1377.922250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1379.261967] Lustre: DEBUG MARKER: Force oleg316-server to discover 192.168.203.16@tcp [ 1380.960144] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1381.299295] Lustre: DEBUG MARKER: Wait for 71811 [ 1382.605992] Lustre: DEBUG MARKER: Finished wait on 71811 [ 1382.864468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1382.905747] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1382.924973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1383.241560] LNet: 72260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.254185] LNetError: 72260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.268513] LNet: Removed LNI 192.168.203.16@tcp [ 1383.692694] Key type .llcrypt unregistered [ 1383.701118] Key type ._llcrypt unregistered [ 1399.802949] Key type ._llcrypt registered [ 1399.807504] Key type .llcrypt registered [ 1401.250509] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:36:13 (1788867373) [ 1401.938308] Key type .llcrypt unregistered [ 1401.947721] Key type ._llcrypt unregistered [ 1407.896171] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1408.026618] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1408.036138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1408.041757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1410.664607] Key type ._llcrypt registered [ 1410.670675] Key type .llcrypt registered [ 1410.860576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.875180] alg: No test for adler32 (adler32-zlib) [ 1411.789039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.847686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.878318] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1412.514203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1412.530877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1420.828541] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:36:33 (1788867393) [ 1420.932471] LNet: 74325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.942942] LNetError: 74325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1420.953881] LNet: Removed LNI 10.1.2.3@tcp [ 1420.987927] LNet: Removed LNI 192.168.203.16@tcp [ 1421.766560] Key type .llcrypt unregistered [ 1421.769599] Key type ._llcrypt unregistered [ 1428.740398] Key type ._llcrypt registered [ 1428.743597] Key type .llcrypt registered [ 1428.981384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1428.991604] alg: No test for adler32 (adler32-zlib) [ 1429.981729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.041462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.098372] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1431.035887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1431.054492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1432.073391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.16@tcp [ 1433.187702] LNetError: Unexpected error -22 connecting to 192.168.203.16@tcp at host 192.168.203.16:988 [ 1438.138164] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:36:51 (1788867411) [ 1438.458336] LNet: 75461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.468489] LNetError: 75461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1438.479355] LNet: Removed LNI 10.1.2.3@tcp [ 1438.494948] LNet: Removed LNI 192.168.203.16@tcp [ 1439.075066] Key type .llcrypt unregistered [ 1439.076877] Key type ._llcrypt unregistered [ 1444.994096] Key type ._llcrypt registered [ 1444.998142] Key type .llcrypt registered [ 1445.314893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1445.356853] alg: No test for adler32 (adler32-zlib) [ 1446.251744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.315588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.337801] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1446.926692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1446.942047] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 1447.700342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.16@tcp1 --nid 192.168.203.16@tcp2 [ 1451.781681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp2 [ 1451.800768] LNetError: 76294:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.16@tcp1 to 192.168.203.16@tcp2 - Route table may be misconfigured [ 1451.826205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp2 192.168.203.16@tcp1 [ 1457.220906] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:37:10 (1788867430) [ 1457.334048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.346680] LNet: 76864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.355720] LNetError: 76864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.363033] LNet: Removed LNI 192.168.203.16@tcp1 [ 1458.417355] LNet: Removed LNI 192.168.203.16@tcp2 [ 1458.451217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.538094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1458.561839] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1459.194309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1459.217686] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1460.114857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1460.143385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.16@tcp [ 1460.188046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp1 [ 1460.232728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp -e network_timeout [ 1460.264170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -e network_timeout [ 1460.299789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -e network_timeout [ 1460.337311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -e network_timeout [ 1460.370813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1460.394596] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1460.405881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1460.486783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1460.571845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1460.578134] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1467.018759] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:37:19 (1788867439) [ 1467.125165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.159313] LNet: 77723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.173705] LNetError: 77723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.181058] LNetError: 77723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1467.187418] LNet: Removed LNI 192.168.203.16@tcp [ 1468.207365] LNet: Removed LNI 192.168.203.16@tcp1 [ 1468.271435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.381514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1468.633310] LNet: 77767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.130812] Key type .llcrypt unregistered [ 1469.134087] Key type ._llcrypt unregistered [ 1479.706591] Key type ._llcrypt registered [ 1479.711557] Key type .llcrypt registered [ 1480.977431] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:37:33 (1788867453) [ 1481.317886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.327605] alg: No test for adler32 (adler32-zlib) [ 1482.191782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.486176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1485.079856] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:37:38 (1788867458) [ 1485.272934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.297134] LNet: 78862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.318532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.387255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.422860] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1486.066813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.105704] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1486.888407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1486.928319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp1 [ 1486.956883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp1 [ 1492.337187] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:37:45 (1788867465) [ 1493.979493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg316-server [ 1495.436361] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:37:48 (1788867468) [ 1497.125334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg316-server [ 1498.574188] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:37:51 (1788867471) [ 1500.062585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg316-server [ 1501.488619] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:37:54 (1788867474) [ 1503.059354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg316-server [ 1504.955461] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:37:57 (1788867477) [ 1506.929574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg316-server [ 1508.439963] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:38:01 (1788867481) [ 1510.244629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg316-server [ 1511.988887] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:38:05 (1788867485) [ 1513.758586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg316-server [ 1515.739722] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:38:08 (1788867488) [ 1517.514098] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg316-server [ 1518.838660] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:38:12 (1788867492) [ 1520.251764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg316-server [ 1521.949693] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:38:14 (1788867494) [ 1522.037738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.071185] LNet: 82734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.082941] LNetError: 82734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.094472] LNet: Removed LNI 192.168.203.16@tcp [ 1522.125660] LNet: Removed LNI 192.168.203.16@tcp1 [ 1522.154163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.215342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.238936] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1522.808823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1522.900266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1523.069654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.099132] LNet: 82885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.110292] LNetError: 82885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.119240] LNetError: 82885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1523.146597] LNet: Removed LNI 192.168.203.16@tcp [ 1523.172038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.233845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1523.272579] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1524.136026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1524.356156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.384315] LNetError: 83030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.415790] LNet: Removed LNI 192.168.203.16@tcp [ 1525.450868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.530847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.565064] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1526.375570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1526.435297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1527.718419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.743856] LNet: 83187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.750158] LNet: 83187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1527.759340] LNetError: 83187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.780694] LNet: Removed LNI 192.168.203.16@tcp [ 1527.828445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.893807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1527.920361] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1528.738640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1528.964456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.032838] LNet: Removed LNI 192.168.203.16@tcp [ 1530.067878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.144324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.192940] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1530.872801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1530.968708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1531.183838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.263822] LNet: Removed LNI 192.168.203.16@tcp [ 1532.298231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.397430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1532.426718] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1533.176200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1533.438367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.467782] LNet: 83624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.474312] LNet: 83624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1533.482607] LNetError: 83624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.488756] LNetError: 83624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1533.505409] LNet: Removed LNI 192.168.203.16@tcp [ 1533.546374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.613954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.652471] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1534.517544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1534.601356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1534.833646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.916844] LNet: Removed LNI 192.168.203.16@tcp [ 1535.945843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.024688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1536.052626] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1536.959217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1537.253733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.346169] LNet: Removed LNI 192.168.203.16@tcp [ 1538.372698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.467568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.531039] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1539.074799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1539.139539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1539.373854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.455107] LNet: Removed LNI 192.168.203.16@tcp [ 1540.486273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.540645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1540.599770] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1541.497270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1543.160479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.179639] LNet: 84219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.187419] LNet: 84219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1543.205719] LNetError: 84219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.214603] LNetError: 84219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1544.236178] LNet: Removed LNI 192.168.203.16@tcp [ 1544.271622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.387116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.417875] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1545.185876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1545.266289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1545.561493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.599727] LNet: Removed LNI 192.168.203.16@tcp [ 1546.627247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.716058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1546.754619] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1547.446259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1547.771644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.839902] LNet: Removed LNI 192.168.203.16@tcp [ 1548.896892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.983381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.016082] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1549.915857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1549.977619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.037447] LNet: Removed LNI 192.168.203.16@tcp [ 1550.060209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.108597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.140173] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1550.867760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1550.969988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.006631] LNet: Removed LNI 192.168.203.16@tcp [ 1551.038475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.100646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1551.181801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.238328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.325400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1551.381868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.407193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.443494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1551.540780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.592679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.661846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1551.699405] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1552.393944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.429622] LNet: Removed LNI 192.168.203.16@tcp [ 1552.452874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.523158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1558.770747] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:38:51 (1788867531) [ 1558.893743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.950093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.035879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1559.056293] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1560.330235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.378958] LNet: 85670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.388573] LNet: 85670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1560.411812] LNetError: 85670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.420243] LNetError: 85670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1560.447500] LNet: Removed LNI 192.168.203.16@tcp231 [ 1560.474671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.554276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1560.603550] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1560.612458] LNet: 85678:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1561.249145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1561.299887] LNet: Removed LNI 192.168.203.16@tcp231 [ 1561.325850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1561.382252] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1562.586872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.615606] LNet: Removed LNI 192.168.203.16@tcp231 [ 1562.636681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.696982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1562.713773] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1562.717992] LNet: 85830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1563.114492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1563.154602] LNet: Removed LNI 192.168.203.16@tcp231 [ 1563.170528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1563.190332] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1564.320683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.358963] LNet: Removed LNI 192.168.203.16@tcp231 [ 1564.383677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.446940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1564.469752] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1564.481447] LNet: 85982:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1564.933620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1564.962472] LNet: Removed LNI 192.168.203.16@tcp231 [ 1564.977829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1565.004291] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1565.953513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.987446] LNet: Removed LNI 192.168.203.16@tcp231 [ 1566.008535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.082473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1566.103518] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1566.559159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1566.598986] LNet: Removed LNI 192.168.203.16@tcp231 [ 1566.622820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1566.668121] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1573.755932] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:39:06 (1788867546) [ 1573.932270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.971769] LNet: Removed LNI 192.168.203.16@tcp231 [ 1574.010171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.102640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1574.147348] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1574.879325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1574.930560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1574.968900] LNet: Removed LNI 192.168.203.16@tcp232 [ 1574.986581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1575.014292] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1575.735754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1575.774081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1575.800052] LNet: Removed LNI 192.168.203.16@tcp232 [ 1575.813859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1575.831747] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1576.478443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1576.508384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1576.536979] LNet: Removed LNI 192.168.203.16@tcp232 [ 1576.553429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1576.576052] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1577.155399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.197370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.221358] LNet: Removed LNI 192.168.203.16@tcp232 [ 1577.234273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1582.777314] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:39:15 (1788867555) [ 1583.678473] Key type .llcrypt unregistered [ 1583.682282] Key type ._llcrypt unregistered [ 1589.868755] Key type ._llcrypt registered [ 1589.870779] Key type .llcrypt registered [ 1590.026975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1590.039730] alg: No test for adler32 (adler32-zlib) [ 1590.981830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1591.020214] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1597.735912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1606.477409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.531264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1606.569991] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1607.727922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1607.773612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1607.815953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1607.871682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1607.903718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1607.944479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1607.995727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1608.029297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1608.062637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1608.082987] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1608.087993] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1608.315380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1608.348470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1608.386640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1608.423970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1608.453470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1608.487270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1608.526495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1608.564233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1608.581684] LNet: 88629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1608.604545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1608.631664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1608.655603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1608.684598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1608.709348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1608.735303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1609.231158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1609.281194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1609.330501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1609.402670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1609.462686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1609.499595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1609.528524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1609.559530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1609.597715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1609.624319] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1609.853865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1609.888262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1609.926751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1609.979563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1610.018263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1610.058946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1610.100212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1610.152808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1610.179798] LNet: 88750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1610.201066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1610.231161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1610.254846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1610.280418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1610.308920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1610.338296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1610.875965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1610.906962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1610.943231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1610.985127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1611.020473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1611.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1611.122922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1611.187951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1611.240994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1611.270259] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1611.562521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1611.610643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1611.640271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1611.681560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1611.748966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1611.784344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1611.833377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1611.867353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1611.889206] LNet: 88870:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.921568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1611.978841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1612.036237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1612.081842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.127384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1612.158551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1612.628918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1612.681666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1612.711386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1612.740548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1612.770509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1612.813214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1612.882972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1612.923182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1612.957352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1613.194260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1613.222846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1613.250513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1613.279438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1613.341390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1613.385309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1613.437806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1613.480625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1613.530277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1613.588118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1613.652719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1613.705670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1613.738935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1613.783547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1614.319746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1614.373754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1614.449104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1614.524420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1614.597986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1614.651421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1614.724144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1614.796150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1614.835941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1614.856338] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1614.862509] LNet: Skipped 1 previous similar message [ 1615.162355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.206108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.244941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.269605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.308972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.373390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.422660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.479717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.521332] LNet: 89111:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1615.528965] LNet: 89111:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1615.551911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.608726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.639365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.683850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1615.726673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1615.791649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.343360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1616.448773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1616.505394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1616.563479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1616.647217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1616.734373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1616.799703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1616.884292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1616.970029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1617.001254] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1617.511247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.565126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1617.610235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.663303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.714607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.778440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.819075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.853911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.885286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.921980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.960740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.994626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.033972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.080241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1623.638252] LNet: 89305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.646161] LNetError: 89305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.656290] LNet: Removed LNI 192.168.203.16@tcp [ 1623.665780] LNet: Removed LNI 192.168.203.16@tcp1 [ 1624.096769] Key type .llcrypt unregistered [ 1624.098770] Key type ._llcrypt unregistered [ 1639.489181] Key type ._llcrypt registered [ 1639.491288] Key type .llcrypt registered [ 1640.473969] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:40:13 (1788867613) [ 1640.889379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1640.909211] alg: No test for adler32 (adler32-zlib) [ 1641.843426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.890087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1641.911182] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1642.679155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1642.726437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1642.754950] LNetError: 90281:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1642.862675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.16@tcp [ 1642.995647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1648.624814] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:40:21 (1788867621) [ 1648.860407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.882449] LNet: 90871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.887844] LNetError: 90871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.901966] LNet: Removed LNI 192.168.203.16@tcp [ 1648.930935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.978191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.006862] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1649.640822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1649.674658] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1650.427760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.16@tcp [ 1650.444384] LNetError: 91137:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1650.977301] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1650.979044] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1651.651843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.16@tcp [ 1659.465828] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:40:32 (1788867632) [ 1661.575836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg316-server [ 1663.217969] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:40:36 (1788867636) [ 1665.118200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg316-server [ 1666.794760] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:40:39 (1788867639) [ 1668.427891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg316-server [ 1669.984991] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:40:43 (1788867643) [ 1670.134142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.168895] LNet: 92764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.184618] LNetError: 92764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.195312] LNet: Removed LNI 192.168.203.16@tcp [ 1670.220370] LNet: Removed LNI 192.168.203.16@tcp1 [ 1670.239640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.293356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.319453] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1670.929110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1670.992847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1671.038977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1671.131596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1671.166314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1671.243346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1677.326367] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:40:50 (1788867650) [ 1677.464313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.510531] LNet: 93491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.524748] LNetError: 93491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.531595] LNetError: 93491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1677.552367] LNet: Removed LNI 192.168.203.16@tcp [ 1677.579256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.683212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.725330] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1685.093669] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:40:57 (1788867657) [ 1685.173021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.222481] LNet: 94193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.239982] LNetError: 94193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.271613] LNet: Removed LNI 192.168.203.16@tcp [ 1685.359287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.500310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.532302] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1694.804753] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:41:07 (1788867667) [ 1697.253951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1699.641661] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:41:12 (1788867672) [ 1700.285706] LNet: 95238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.294309] LNetError: 95238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.313974] LNet: Removed LNI 192.168.203.16@tcp [ 1700.920771] Key type .llcrypt unregistered [ 1700.923826] Key type ._llcrypt unregistered [ 1708.981882] Key type ._llcrypt registered [ 1708.984883] Key type .llcrypt registered [ 1709.330137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1709.354966] alg: No test for adler32 (adler32-zlib) [ 1710.331356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1710.388983] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1718.561056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1723.226857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.116@tcp [ 1723.708652] LNet: 95654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.717119] LNetError: 95654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.739522] LNet: Removed LNI 192.168.203.16@tcp [ 1724.253728] Key type .llcrypt unregistered [ 1724.261516] Key type ._llcrypt unregistered [ 1737.776968] Key type ._llcrypt registered [ 1737.780894] Key type .llcrypt registered [ 1739.451974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1741.442541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1743.368706] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:41:56 (1788867716) [ 1745.060906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1746.803449] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:41:59 (1788867719) [ 1748.837869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg316-server [ 1750.224959] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:42:03 (1788867723) [ 1753.017300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg316-server [ 1754.788200] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:42:07 (1788867727) [ 1756.664131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.679417] alg: No test for adler32 (adler32-zlib) [ 1757.891475] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1758.261111] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1760.000425] Key type lgssc registered [ 1761.457879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1783.666217] Key type lgssc unregistered [ 1783.908712] LNet: 98827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.920422] LNetError: 98827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.936732] LNet: Removed LNI 192.168.203.16@tcp [ 1784.536539] Key type .llcrypt unregistered [ 1784.539733] Key type ._llcrypt unregistered [ 1797.619522] Key type ._llcrypt registered [ 1797.621564] Key type .llcrypt registered [ 1799.132068] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:42:51 (1788867771) [ 1799.413826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1799.427277] alg: No test for adler32 (adler32-zlib) [ 1800.372655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.476965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1800.511849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1800.597043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1800.647477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1806.680688] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:42:59 (1788867779) [ 1806.964480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.988789] LNet: 100240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.009960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.062694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.083388] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1808.007549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.030746] LNet: 100384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.039714] LNetError: 100384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.056083] LNet: Removed LNI 192.168.203.16@tcp [ 1808.091365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.139391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1808.152836] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1808.680788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.707777] LNetError: 100412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.719809] LNet: Removed LNI 192.168.203.16@tcp [ 1808.740728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.824663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1808.864513] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1816.905570] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:43:09 (1788867789) [ 1817.028374] LNet: 101001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.033402] LNet: 101001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1817.048966] LNetError: 101001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.071395] LNet: Removed LNI 192.168.203.16@tcp [ 1817.540266] Key type .llcrypt unregistered [ 1817.542576] Key type ._llcrypt unregistered [ 1824.894357] Key type ._llcrypt registered [ 1824.896216] Key type .llcrypt registered [ 1825.130439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1825.142251] alg: No test for adler32 (adler32-zlib) [ 1826.054352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.150471] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1826.154377] LNet: *** cfs_fail_loc=e003, val=100*** [ 1826.981264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1826.999184] LNet: *** cfs_fail_loc=e003, val=100*** [ 1827.002353] LNet: Skipped 2 previous similar messages [ 1827.008664] Ping reply received with mapping: [ 1827.011188] found metadata reply NI[192.168.203.16@tcp] [ 1827.014610] 10 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1827.021784] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1827.028140] 5a 5a 5a 5a 5a 5a 5a 5a [ 1827.237507] LNet: 101399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.259586] LNetError: 101399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.287166] LNet: Removed LNI 192.168.203.16@tcp [ 1828.328514] Key type .llcrypt unregistered [ 1828.332607] Key type ._llcrypt unregistered [ 1840.482781] Key type ._llcrypt registered [ 1840.484633] Key type .llcrypt registered [ 1841.652699] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:43:34 (1788867814) [ 1842.159363] Key type .llcrypt unregistered [ 1842.162436] Key type ._llcrypt unregistered [ 1848.441554] Key type ._llcrypt registered [ 1848.443965] Key type .llcrypt registered [ 1848.648909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1848.664726] alg: No test for adler32 (adler32-zlib) [ 1849.684566] LNetError: 102436:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1850.061430] Key type .llcrypt unregistered [ 1850.063249] Key type ._llcrypt unregistered [ 1855.135565] Key type ._llcrypt registered [ 1855.140633] Key type .llcrypt registered [ 1856.460615] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:43:49 (1788867829) [ 1856.989523] Key type .llcrypt unregistered [ 1856.993493] Key type ._llcrypt unregistered [ 1863.486659] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1863.496271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1865.096549] Key type ._llcrypt registered [ 1865.100061] Key type .llcrypt registered [ 1865.380718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1865.403308] alg: No test for adler32 (adler32-zlib) [ 1866.334439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.436112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1877.527932] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:44:09 (1788867849) [ 1877.676746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.711520] LNet: 104196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.726478] LNetError: 104196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1877.756695] LNet: Removed LNI 10.1.2.3@tcp [ 1877.769609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.913626] Key type .llcrypt unregistered [ 1878.915499] Key type ._llcrypt unregistered [ 1888.126572] Key type ._llcrypt registered [ 1888.129757] Key type .llcrypt registered [ 1888.402304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1888.421205] alg: No test for adler32 (adler32-zlib) [ 1889.315789] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1889.403653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1889.755038] LNet: 104488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.778234] LNetError: 104488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.809651] LNet: Removed LNI 192.168.203.16@tcp [ 1890.976378] Key type .llcrypt unregistered [ 1890.982635] Key type ._llcrypt unregistered [ 1903.184485] Key type ._llcrypt registered [ 1903.192706] Key type .llcrypt registered [ 1904.799643] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:44:37 (1788867877) [ 1905.961545] Key type .llcrypt unregistered [ 1905.965191] Key type ._llcrypt unregistered [ 1914.838558] Key type ._llcrypt registered [ 1914.840343] Key type .llcrypt registered [ 1915.110589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1915.143094] alg: No test for adler32 (adler32-zlib) [ 1922.494830] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:44:55 (1788867895) [ 1922.767624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.813541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.886448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.909981] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1923.611868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1923.640949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1923.674806] LNet: Removed LNI 192.168.203.16@tcp [ 1923.967127] LNet: 106229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.446884] Key type .llcrypt unregistered [ 1924.448665] Key type ._llcrypt unregistered [ 1935.585174] Key type ._llcrypt registered [ 1935.586854] Key type .llcrypt registered [ 1936.836142] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:45:09 (1788867909) [ 1937.585238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.604487] alg: No test for adler32 (adler32-zlib) [ 1938.571917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.653936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.686069] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1939.480266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1939.541441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.16@tcp [ 1939.575901] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1939.580963] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1939.584390] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1939, 0, 0 [ 1939.590070] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1939.594091] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1939.597547] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1939.603309] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1939, 0, 0 [ 1939.608021] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1939.611201] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1939.616410] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1939.619287] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1939, 0, 1989 [ 1939.628762] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1939.632603] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1939.636081] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1939.638957] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1939, 0, 1989 [ 1939.644359] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1946.139120] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:45:19 (1788867919) [ 1946.631551] LNet: 107765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.637429] LNetError: 107765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.688758] LNet: Removed LNI 192.168.203.16@tcp [ 1948.294680] Key type .llcrypt unregistered [ 1948.298414] Key type ._llcrypt unregistered [ 1954.685996] Key type ._llcrypt registered [ 1954.691257] Key type .llcrypt registered [ 1954.950842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1954.969106] alg: No test for adler32 (adler32-zlib) [ 1955.883601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1955.918075] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1962.557682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1971.895774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.974964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1972.007330] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1972.997908] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1973.214731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1973.226263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1973.232600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1975.590208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1975.618866] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1984.749495] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:45:57 (1788867957) [ 1984.913105] LNet: 109267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.924241] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.933434] LNet: Removed LNI 192.168.203.16@tcp [ 1985.952291] LNet: Removed LNI 192.168.203.16@tcp1 [ 1985.957761] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1985.973696] LNetError: 109267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1985.999967] LNet: Removed LNI 10.1.2.3@tcp99 [ 1986.642341] Key type .llcrypt unregistered [ 1986.653431] Key type ._llcrypt unregistered [ 1993.085744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1993.090119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1995.547212] Key type ._llcrypt registered [ 1995.551183] Key type .llcrypt registered [ 1995.711158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1995.736640] alg: No test for adler32 (adler32-zlib) [ 1996.714909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1996.786418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1996.823831] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1997.505037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1997.523668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2000.048442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2000.118612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2000.184632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2000.242299] LNetError: 110014:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2000.277683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2000.338640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2001.004194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2001.103359] LNet: 110034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.114253] LNetError: 110034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2001.120797] LNet: Removed LNI 10.1.2.3@tcp [ 2002.151035] LNet: Removed LNI 192.168.203.16@tcp [ 2002.885680] Key type .llcrypt unregistered [ 2002.887835] Key type ._llcrypt unregistered [ 2016.154620] Key type ._llcrypt registered [ 2016.157050] Key type .llcrypt registered [ 2017.694950] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:46:30 (1788867990) [ 2018.123390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2018.148310] alg: No test for adler32 (adler32-zlib) [ 2019.054202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.153980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.235683] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2019.941996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2020.009440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg316-client@tcp [ 2026.915596] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:46:39 (1788867999) [ 2027.192366] LNet: 111565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2027.209183] LNetError: 111565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.264100] LNet: Removed LNI 192.168.203.16@tcp [ 2028.875451] Key type .llcrypt unregistered [ 2028.879339] Key type ._llcrypt unregistered [ 2035.149307] Key type ._llcrypt registered [ 2035.152101] Key type .llcrypt registered [ 2035.382914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.404291] alg: No test for adler32 (adler32-zlib) [ 2036.406269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2036.449457] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2036.509752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2043.267186] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:46:55 (1788868015) [ 2044.651672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2046.629678] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:46:59 (1788868019) [ 2048.171984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2049.673332] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:47:02 (1788868022) [ 2049.962722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.986331] LNet: 112987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.002198] LNetError: 112987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.017877] LNet: Removed LNI 192.168.203.16@tcp [ 2050.049908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.114352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.148644] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2050.907670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2050.980953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2057.088814] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:47:09 (1788868029) [ 2057.214140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.251670] LNet: 113689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.258150] LNetError: 113689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.271424] LNet: Removed LNI 192.168.203.16@tcp [ 2057.285026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.369775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2057.399941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2058.166281] Key type .llcrypt unregistered [ 2058.171083] Key type ._llcrypt unregistered [ 2069.976384] Key type ._llcrypt registered [ 2069.978346] Key type .llcrypt registered [ 2071.710282] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:47:24 (1788868044) [ 2072.281602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2072.308496] alg: No test for adler32 (adler32-zlib) [ 2073.308079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.383557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2073.422213] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2074.000564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2074.060918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2080.650570] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:47:33 (1788868053) [ 2080.736214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2080.778586] LNet: 115242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.790543] LNetError: 115242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.823675] LNet: Removed LNI 192.168.203.16@tcp [ 2080.880049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.929813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2080.958763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2087.183361] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:47:40 (1788868060) [ 2087.277159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.323120] LNet: 115822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.354519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.471682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2087.935722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2087.979079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2088.065194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2088.537906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2088.569947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2088.653906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.675063] LNet: 115861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.700284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.768740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2089.287504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2089.335996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2089.407614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2089.781861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2089.809714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2096.238833] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:47:49 (1788868069) [ 2096.521933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.565332] LNet: 116461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.620690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.726843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.767279] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2097.392733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2097.436680] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2098.385854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2098.474052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2098.725837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.771906] LNetError: 116741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.808184] LNet: Removed LNI 192.168.203.16@tcp [ 2099.812652] LNetError: 116741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.844575] LNet: Removed LNI 192.168.203.16@tcp2 [ 2099.884377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.965021] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2099.979397] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2099.999229] LNet: 116747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2100.678881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2100.914419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.937706] LNet: 116784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.947088] LNet: 116784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2101.984514] LNet: Removed LNI 192.168.203.16@tcp [ 2101.986950] LNetError: 116784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.004033] LNetError: 116784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2102.028978] LNet: Removed LNI 192.168.203.16@tcp2 [ 2102.068398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.131443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.163145] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2102.879618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2102.917244] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2103.833369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2103.910349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2103.996335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2104.285288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.376086] LNet: Removed LNI 192.168.203.16@tcp [ 2106.407695] LNet: Removed LNI 192.168.203.16@tcp2 [ 2106.490746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.600434] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2106.613894] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2106.623764] LNet: 117075:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2107.294651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2107.542823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.572474] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.581767] LNetError: 117112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2107.594145] LNet: Removed LNI 192.168.203.16@tcp [ 2108.647161] LNet: Removed LNI 192.168.203.16@tcp2 [ 2108.699658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.787597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.807458] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2109.619606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2109.641860] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2110.410985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2110.474856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2110.686095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2110.705897] LNet: 117392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.715669] LNet: 117392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2111.776329] LNet: Removed LNI 192.168.203.16@tcp [ 2111.783771] LNet: Removed LNI 192.168.203.16@tcp2 [ 2111.809510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.894629] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2111.902435] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2111.911650] LNet: 117398:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2112.291267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2112.479731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.504173] LNet: Removed LNI 192.168.203.16@tcp [ 2113.516205] LNet: Removed LNI 192.168.203.16@tcp2 [ 2113.541676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.610093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.636921] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2114.577912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2114.607754] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2115.596598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2115.657585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2115.737695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2115.992519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.007355] LNetError: 117719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.012360] LNetError: 117719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2116.017713] LNet: Removed LNI 192.168.203.16@tcp [ 2117.030645] LNet: Removed LNI 192.168.203.16@tcp2 [ 2117.076703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.143692] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2117.163479] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2117.169518] LNet: 117726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.928410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2118.245078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.286650] LNet: Removed LNI 192.168.203.16@tcp [ 2119.338144] LNet: Removed LNI 192.168.203.16@tcp2 [ 2119.403312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.475203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.498619] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2120.238786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.271161] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2121.443563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2121.521362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2121.585521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2121.936636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2122.976231] LNet: Removed LNI 192.168.203.16@tcp [ 2122.990940] LNet: Removed LNI 192.168.203.16@tcp2 [ 2123.017568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.127145] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2123.137989] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2123.146395] LNet: 118066:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2123.964541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2124.323165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.408223] LNet: Removed LNI 192.168.203.16@tcp [ 2125.420127] LNet: Removed LNI 192.168.203.16@tcp2 [ 2125.459283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.502831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.532690] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2126.294523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.312949] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2127.204856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2127.265370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2127.314879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2127.413724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2127.719911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.739470] LNet: 118419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.743751] LNet: 118419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2128.800300] LNet: Removed LNI 192.168.203.16@tcp [ 2129.847895] LNet: Removed LNI 192.168.203.16@tcp2 [ 2129.966058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.069985] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2130.078452] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2130.760826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2138.103850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2139.571868] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:48:32 (1788868112) [ 2139.677073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.711799] LNetError: 119179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.721811] LNetError: 119179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2139.735342] LNet: Removed LNI 192.168.203.16@tcp [ 2139.752508] LNet: Removed LNI 192.168.203.16@tcp2 [ 2139.789455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.260840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.304545] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2140.995718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2141.014671] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2142.022167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.061554] LNet: Removed LNI 10.1.2.3@tcp [ 2142.072038] LNet: Removed LNI 192.168.203.16@tcp [ 2142.096671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.183752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2142.229591] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2142.249837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2142.680797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2142.717213] LNet: Removed LNI 192.168.203.16@tcp [ 2142.727724] LNet: Removed LNI 10.1.2.3@tcp [ 2148.856196] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:48:41 (1788868121) [ 2148.952054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.990452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.070580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.099545] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2149.703612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2149.746979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.791962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.827826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.863947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.897836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.954635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2149.995299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2150.029992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2150.077645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2150.102836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2150.234167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2156.328229] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:48:49 (1788868129) [ 2156.532409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.606187] LNet: Removed LNI 192.168.203.16@tcp [ 2157.634248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.741805] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2157.883796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2157.893268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2157.904957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2160.200989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.234161] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2160.825817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2160.846388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.831099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2162.885511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2162.957244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.16@tcp -m PUT -e local_timeout -r 1 [ 2163.013429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2163.072122] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2163.074667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2163.309767] LNetError: 120866:0:(lib-move.c:4801:lnet_parse()) 192.168.203.16@tcp, src 192.168.203.16@tcp: Dropping PUT (error -108 looking up sender) [ 2164.324892] LNet: 121325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.334068] LNet: 121325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2164.345395] LNet: Removed LNI 10.1.2.3@tcp [ 2165.423908] LNet: Removed LNI 192.168.203.16@tcp [ 2165.863740] Key type .llcrypt unregistered [ 2165.865877] Key type ._llcrypt unregistered [ 2177.910275] Key type ._llcrypt registered [ 2177.913118] Key type .llcrypt registered [ 2179.646713] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:49:12 (1788868152) [ 2180.286823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.330152] alg: No test for adler32 (adler32-zlib) [ 2181.472333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.230860] LNet: 122174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.875755] Key type .llcrypt unregistered [ 2182.879339] Key type ._llcrypt unregistered [ 2190.255560] Key type ._llcrypt registered [ 2190.257987] Key type .llcrypt registered [ 2190.467969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2190.504127] alg: No test for adler32 (adler32-zlib) [ 2191.496730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2191.518859] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2200.054392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2200.289730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2200.979150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2201.264508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2201.322092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2201.366624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2201.424319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2201.498630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2201.538349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2201.578457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2201.620109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2201.645621] LNet: 122621:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2201.671592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2201.729789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2201.760384] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2201.785493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2201.813949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2201.878931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2201.933154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2202.704100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2202.939389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2202.965475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2202.999689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.030300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2203.056568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2203.084703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2203.115143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2203.146756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2203.165129] LNet: 122738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2203.190259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2203.238684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2203.299269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2203.334870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2203.367243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2203.397337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.155493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2204.565219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2204.597100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2204.646667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2204.689884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2204.755734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2204.810389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2204.864460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2204.925404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2204.948410] LNet: 122856:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2204.979692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2205.038310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2205.082758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2205.119863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2205.163807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2205.193736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2205.942486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2206.302129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2206.352456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2206.408637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2206.466610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2206.505545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2206.565222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2206.595969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2206.628336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2206.661258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2206.691394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2206.730723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2206.766990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2206.824435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2206.863506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2207.211629] LNet: 123019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.216877] LNetError: 123019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.232629] LNet: Removed LNI 192.168.203.16@tcp [ 2207.702943] Key type .llcrypt unregistered [ 2207.707578] Key type ._llcrypt unregistered [ 2229.674381] Key type ._llcrypt registered [ 2229.678494] Key type .llcrypt registered [ 2231.509238] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:50:03 (1788868203) [ 2232.648508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.693480] alg: No test for adler32 (adler32-zlib) [ 2233.832452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.594334] LNet: 123904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.907406] Key type .llcrypt unregistered [ 2235.912284] Key type ._llcrypt unregistered [ 2244.941662] Key type ._llcrypt registered [ 2244.945193] Key type .llcrypt registered [ 2245.096598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2245.115091] alg: No test for adler32 (adler32-zlib) [ 2246.091115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2246.152536] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2254.134434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2263.452273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.484808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2263.523716] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 2264.443704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2264.916660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2265.168897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2265.197878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2265.230335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2265.271922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2265.311616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2265.363278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2265.396333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2265.433290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2265.464185] LNet: 124553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2265.482368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2265.507737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2265.532979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2265.573662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2265.632273] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2265.637165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2265.693140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2266.363385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2266.652829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2266.682715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2266.706847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2266.734276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2266.764443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2266.791646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2266.816626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2266.842224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2266.865899] LNet: 124671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2266.883739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2266.910338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2266.935737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2266.966642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2267.012765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2267.060892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2267.605924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2267.826868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2267.851625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2267.888966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2267.915807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2267.940354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2267.980937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2268.014559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2268.039153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2268.053909] LNet: 124788:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2268.073930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2268.102934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2268.138439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2268.160983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2268.193706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2268.248896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2268.865189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2269.172678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.208296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2269.240584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2269.285471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2269.313626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2269.358217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2269.409597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2269.447395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2269.475602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2269.505394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2269.534957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2269.562583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2269.591525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2269.623492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.655622] LNet: 124986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.667396] LNetError: 124986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.678477] LNet: Removed LNI 192.168.203.16@tcp [ 2274.695224] LNet: Removed LNI 192.168.203.16@tcp1 [ 2275.185755] Key type .llcrypt unregistered [ 2275.196480] Key type ._llcrypt unregistered [ 2292.307079] Key type ._llcrypt registered [ 2292.308536] Key type .llcrypt registered [ 2293.973552] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:51:06 (1788868266) [ 2294.425875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.443744] alg: No test for adler32 (adler32-zlib) [ 2295.475255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.579833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.603545] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2296.616058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2296.651382] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2297.816280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2297.929073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2298.976312] LNet: Removed LNI 192.168.203.16@tcp [ 2306.730360] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:51:19 (1788868279) [ 2308.748252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2310.934366] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:51:23 (1788868283) [ 2318.216901] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:51:31 (1788868291) [ 2325.061426] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:51:38 (1788868298) [ 2332.135414] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:51:45 (1788868305) [ 2338.795498] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:51:51 (1788868311) [ 2345.202115] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:51:58 (1788868318) [ 2353.419856] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:52:06 (1788868326) [ 2361.200182] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:52:13 (1788868333) [ 2368.726499] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:52:21 (1788868341) [ 2375.879877] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:52:28 (1788868348) [ 2382.376720] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:52:35 (1788868355) [ 2388.681114] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:52:41 (1788868361) [ 2395.921155] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:52:48 (1788868368) [ 2402.359952] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:52:55 (1788868375) [ 2409.901209] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:53:02 (1788868382) [ 2417.580335] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:53:10 (1788868390) [ 2427.987805] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:53:19 (1788868399) [ 2436.798750] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:53:29 (1788868409) [ 2436.926968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.972781] LNet: 136672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.990404] LNetError: 136672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.024818] LNet: Removed LNI 192.168.203.16@tcp2 [ 2437.074433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.175704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2437.199237] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2438.265226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.306837] LNet: 136820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.324300] LNetError: 136820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.361802] LNet: Removed LNI 192.168.203.16@tcp [ 2438.383579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.488547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2438.522455] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2447.323683] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:53:39 (1788868419) [ 2455.513975] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:53:48 (1788868428) [ 2464.201677] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:53:56 (1788868436) [ 2473.160974] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:54:05 (1788868445) [ 2480.985368] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:54:13 (1788868453) [ 2489.398865] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:54:21 (1788868461) [ 2497.031308] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:54:29 (1788868469) [ 2503.279673] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:54:36 (1788868476) [ 2510.332668] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:54:43 (1788868483) [ 2519.425590] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:54:51 (1788868491) [ 2527.439112] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:55:00 (1788868500) [ 2534.454927] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:55:07 (1788868507) [ 2542.890348] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2296 sec ========== 07:55:15 (1788868515) [ 2543.312388] LNet: 144292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.323142] LNetError: 144292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.351744] LNet: Removed LNI 192.168.203.16@tcp [ 2544.148586] Key type .llcrypt unregistered [ 2544.155988] Key type ._llcrypt unregistered [ 2556.742112] Key type ._llcrypt registered [ 2556.744657] Key type .llcrypt registered [ 2557.137119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2557.159718] alg: No test for adler32 (adler32-zlib) [ 2558.205350] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2558.475686] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2560.128189] Key type lgssc registered [ 2561.272393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.774477] Lustre: Mounted lustre-client - version 2.17.57_104_g1a15c94 [ 2633.562949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2640.249483] Lustre: Unmounted lustre-client [ 2661.992758] Key type lgssc unregistered [ 2662.370063] LNet: 147241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.378433] LNetError: 147241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.414176] LNet: Removed LNI 192.168.203.16@tcp [ 2663.002310] Key type .llcrypt unregistered [ 2663.011201] Key type ._llcrypt unregistered