[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 438365073 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007576] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.011028] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013044] SELinux: Initializing. [ 0.015019] *** VALIDATE selinux *** [ 0.023687] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028138] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029145] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031079] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032101] *** VALIDATE tmpfs *** [ 0.034108] *** VALIDATE proc *** [ 0.035261] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037275] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040038] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044265] debug: unmapping init [mem 0xffffffff9ce59000-0xffffffff9ce60fff] [ 0.046203] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047756] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050028] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053022] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055289] rcu: Hierarchical SRCU implementation. [ 0.057564] smp: Bringing up secondary CPUs ... [ 0.058628] x86: Booting SMP configuration: [ 0.059032] .... node #0, CPUs: #1 #2 #3 [ 0.064368] smp: Brought up 1 node, 4 CPUs [ 0.066013] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105482] node 0 deferred pages initialised in 37ms [ 0.108596] devtmpfs: initialized [ 0.110157] x86/mm: Memory block size: 128MB [ 0.112168] gcov: version magic: 0x41383552 [ 0.113611] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119318] pinctrl core: initialized pinctrl subsystem [ 0.122160] [ 0.122817] ************************************************************* [ 0.125013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127013] ** ** [ 0.129052] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.132013] ** ** [ 0.134013] ** This means that this kernel is built to expose internal ** [ 0.137013] ** IOMMU data structures, which may compromise security on ** [ 0.139013] ** your system. ** [ 0.142011] ** ** [ 0.144012] ** If you see this message and you are not debugging the ** [ 0.146012] ** kernel, report this immediately to your vendor! ** [ 0.149012] ** ** [ 0.151014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153040] ************************************************************* [ 0.156696] NET: Registered protocol family 16 [ 0.158489] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166416] cpuidle: using governor menu [ 0.167567] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.168562] PCI: Using configuration type 1 for base access [ 0.170096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.177177] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.178012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.179168] cryptd: max_cpu_qlen set to 1000 [ 0.181061] ACPI: Added _OSI(Module Device) [ 0.182008] ACPI: Added _OSI(Processor Device) [ 0.182885] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.184010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.187319] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.191236] ACPI: Interpreter enabled [ 0.192057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.194011] ACPI: Using IOAPIC for interrupt routing [ 0.196084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.200440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.211130] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.216021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.219099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.224874] acpiphp: Slot [2] registered [ 0.227120] acpiphp: Slot [5] registered [ 0.229116] acpiphp: Slot [6] registered [ 0.230116] acpiphp: Slot [3] registered [ 0.232173] acpiphp: Slot [4] registered [ 0.234173] acpiphp: Slot [7] registered [ 0.235115] acpiphp: Slot [8] registered [ 0.237146] acpiphp: Slot [9] registered [ 0.238097] acpiphp: Slot [10] registered [ 0.239130] acpiphp: Slot [11] registered [ 0.241089] acpiphp: Slot [12] registered [ 0.242108] acpiphp: Slot [13] registered [ 0.244103] acpiphp: Slot [14] registered [ 0.245129] acpiphp: Slot [15] registered [ 0.247093] acpiphp: Slot [16] registered [ 0.249116] acpiphp: Slot [17] registered [ 0.250148] acpiphp: Slot [18] registered [ 0.252102] acpiphp: Slot [19] registered [ 0.254109] acpiphp: Slot [20] registered [ 0.255135] acpiphp: Slot [21] registered [ 0.257103] acpiphp: Slot [22] registered [ 0.258098] acpiphp: Slot [23] registered [ 0.260083] acpiphp: Slot [24] registered [ 0.261080] acpiphp: Slot [25] registered [ 0.263107] acpiphp: Slot [26] registered [ 0.265131] acpiphp: Slot [27] registered [ 0.267169] acpiphp: Slot [28] registered [ 0.268076] acpiphp: Slot [29] registered [ 0.270096] acpiphp: Slot [30] registered [ 0.271117] acpiphp: Slot [31] registered [ 0.273077] PCI host bridge to bus 0000:00 [ 0.274018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.284020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291026] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.293317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.299998] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.303836] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.307019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.312491] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.321889] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.333010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.342230] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.347014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.353013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.361012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.370150] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.375017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.383031] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.396018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.404310] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.407472] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.412386] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.415419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.418231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.424665] iommu: Default domain type: Passthrough [ 0.425000] SCSI subsystem initialized [ 0.428141] ACPI: bus type USB registered [ 0.429094] usbcore: registered new interface driver usbfs [ 0.431083] usbcore: registered new interface driver hub [ 0.432056] usbcore: registered new device driver usb [ 0.434152] pps_core: LinuxPPS API ver. 1 registered [ 0.435006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.437044] PTP clock support registered [ 0.439106] EDAC MC: Ver: 3.0.0 [ 0.440413] PCI: Using ACPI for IRQ routing [ 0.443185] NetLabel: Initializing [ 0.444015] NetLabel: domain hash size = 128 [ 0.445015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.447111] NetLabel: unlabeled traffic allowed by default [ 0.449159] vgaarb: loaded [ 0.450261] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.452010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.457603] clocksource: Switched to clocksource kvm-clock [ 0.569124] VFS: Disk quotas dquot_6.6.0 [ 0.570842] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.573372] *** VALIDATE ramfs *** [ 0.574599] *** VALIDATE hugetlbfs *** [ 0.576937] pnp: PnP ACPI init [ 0.579573] pnp: PnP ACPI: found 6 devices [ 0.597839] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.601464] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.603904] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.606269] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.608935] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.611457] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.614539] NET: Registered protocol family 2 [ 0.617218] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.622279] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.626181] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.631093] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.634624] TCP: Hash tables configured (established 65536 bind 65536) [ 0.637956] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.641428] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.644705] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.647854] NET: Registered protocol family 1 [ 0.650203] RPC: Registered named UNIX socket transport module. [ 0.652855] RPC: Registered udp transport module. [ 0.654241] RPC: Registered tcp transport module. [ 0.655403] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.657217] NET: Registered protocol family 44 [ 0.658290] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.659721] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.661400] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.663201] PCI: CLS 0 bytes, default 64 [ 0.665438] Unpacking initramfs... [ 2.117222] debug: unmapping init [mem 0xffff9020bcc64000-0xffff9020bffcffff] [ 2.123819] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.126404] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.131080] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.629140] Initialise system trusted keyrings [ 2.631083] Key type blacklist registered [ 2.633352] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.646322] zbud: loaded [ 2.649156] *** VALIDATE nfs *** [ 2.650257] *** VALIDATE nfs4 *** [ 2.651763] pstore: using deflate compression [ 2.655251] Platform Keyring initialized [ 2.785276] NET: Registered protocol family 38 [ 2.787388] Key type asymmetric registered [ 2.789240] Asymmetric key parser 'x509' registered [ 2.791386] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.794668] io scheduler mq-deadline registered [ 2.796480] io scheduler kyber registered [ 2.798393] io scheduler bfq registered [ 2.800335] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.803711] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.806949] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.810341] ACPI: Power Button [PWRF] [ 2.817958] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.824809] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.833781] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.861729] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.888867] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.894278] Non-volatile memory driver v1.3 [ 2.896091] Linux agpgart interface v0.103 [ 2.928801] virtio_blk virtio1: [vda] 146160 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.933600] vda: detected capacity change from 0 to 74833920 [ 2.948900] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.952415] vdb: detected capacity change from 0 to 1073741824 [ 2.959304] libphy: Fixed MDIO Bus: probed [ 2.964792] usbcore: registered new interface driver usbserial_generic [ 2.967583] usbserial: USB Serial support registered for generic [ 2.969990] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.975111] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.977339] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.980197] mousedev: PS/2 mouse device common for all mice [ 2.983399] rtc_cmos 00:05: RTC can wake from S4 [ 2.985770] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.987406] rtc_cmos 00:05: registered as rtc0 [ 2.992625] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.995708] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.996273] intel_pstate: CPU model not supported [ 3.002106] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.005657] hid: raw HID events driver (C) Jiri Kosina [ 3.008022] usbcore: registered new interface driver usbhid [ 3.009909] usbhid: USB HID core driver [ 3.011325] drop_monitor: Initializing network drop monitor service [ 3.013473] Initializing XFRM netlink socket [ 3.015329] NET: Registered protocol family 10 [ 3.018499] Segment Routing with IPv6 [ 3.020170] NET: Registered protocol family 17 [ 3.022470] mpls_gso: MPLS GSO support [ 3.030080] RAS: Correctable Errors collector initialized. [ 3.032689] AVX version of gcm_enc/dec engaged. [ 3.034732] AES CTR mode by8 optimization enabled [ 3.119477] sched_clock: Marking stable (3119279018, 0)->(4041666809, -922387791) [ 3.122324] registered taskstats version 1 [ 3.124350] Loading compiled-in X.509 certificates [ 3.127223] zswap: loaded using pool lzo/zbud [ 3.154423] Key type big_key registered [ 3.167402] Key type encrypted registered [ 3.168763] ima: No TPM chip found, activating TPM-bypass! [ 3.170789] ima: Allocated hash algorithm: sha1 [ 3.172396] ima: No architecture policies found [ 3.174095] evm: Initialising EVM extended attributes: [ 3.175898] evm: security.selinux [ 3.177193] evm: security.ima [ 3.178365] evm: security.capability [ 3.179769] evm: HMAC attrs: 0x1 [ 3.182287] rtc_cmos 00:05: setting system clock to 2026-08-19 10:57:10 UTC (1787137030) [ 3.188579] debug: unmapping init [mem 0xffffffff9de03000-0xffffffff9dffffff] [ 3.192057] debug: unmapping init [mem 0xffffffff9cb82000-0xffffffff9ce58fff] [ 3.199131] Write protecting the kernel read-only data: 28672k [ 3.202217] debug: unmapping init [mem 0xffffffff9b203000-0xffffffff9b3fffff] [ 3.204773] debug: unmapping init [mem 0xffffffff9bb14000-0xffffffff9bbfffff] [ 3.248325] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.256607] systemd[1]: Detected virtualization kvm. [ 3.258690] systemd[1]: Detected architecture x86-64. [ 3.261103] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.288608] systemd[1]: No hostname configured. [ 3.290671] systemd[1]: Set hostname to . [ 3.293215] random: systemd: uninitialized urandom read (16 bytes read) [ 3.296237] systemd[1]: Initializing machine ID from random generator. [ 3.342472] random: ln: uninitialized urandom read (6 bytes read) [ 3.447592] random: systemd: uninitialized urandom read (16 bytes read) [ 3.450623] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.455833] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.459714] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.159802] device-mapper: uevent: version 1.0.3 [ 4.162754] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.904359] virtio_net virtio0 ens2: renamed from eth0 [ 4.983190] scsi host0: ata_piix [ 5.043379] scsi host1: ata_piix [ 5.045176] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.047806] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.297312] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.685467] random: crng init done [ 9.686662] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.132919] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.299415] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.515914] SELinux: Disabled at runtime. [ 11.577567] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.586336] systemd[1]: Detected virtualization kvm. [ 11.588395] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.073174] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.076183] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.079394] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.081801] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.084178] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.094074] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.100085] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ 12.292144] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.535933] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.820977] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.846761] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.994814] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.011970] EDAC sbridge: Ver: 1.1.2 [ 14.140131] Key type dns_resolver registered [ 14.445431] NFS: Registering the id_resolver key type [ 14.447426] Key type id_resolver registered [ 14.448897] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-client login: [ 71.534733] libcfs: loading out-of-tree module taints kernel. [ 71.822451] Key type ._llcrypt registered [ 71.828628] Key type .llcrypt registered [ 72.289219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 72.313204] alg: No test for adler32 (adler32-zlib) [ 73.808797] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 74.744241] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 76.583232] Key type lgssc registered [ 78.812914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 209.156452] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8 [ 213.974561] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.589491] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing check_logdir /tmp/testlogs/ [ 232.186536] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing yml_node [ 234.975159] Lustre: lustre-OST0000-osc-ffff902109181000: disconnect after 24s idle [ 236.759984] Lustre: DEBUG MARKER: Client: 2.17.57.8 [ 239.129286] Lustre: DEBUG MARKER: MDS: 2.17.57.8 [ 241.235280] Lustre: DEBUG MARKER: OSS: 2.17.57.8 [ 242.818869] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 19 07:01:08 EDT 2026 [ 257.647591] Lustre: Unmounted lustre-client [ 287.406651] Key type lgssc unregistered [ 287.766936] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.773751] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 287.793429] LNet: Removed LNI 192.168.201.51@tcp [ 289.067177] Key type .llcrypt unregistered [ 289.070548] Key type ._llcrypt unregistered [ 316.896323] Key type ._llcrypt registered [ 316.900817] Key type .llcrypt registered [ 317.152600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 317.160863] alg: No test for adler32 (adler32-zlib) [ 318.029284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 318.091669] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 318.101481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 321.285380] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 321.501132] LNet: 9307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.519351] LNetError: 9307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 321.557363] LNet: Removed LNI 192.168.201.51@tcp [ 322.241382] Key type .llcrypt unregistered [ 322.245804] Key type ._llcrypt unregistered [ 328.838442] Key type ._llcrypt registered [ 328.842713] Key type .llcrypt registered [ 330.087499] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:02:35 (1787137355) [ 330.484273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 330.529621] alg: No test for adler32 (adler32-zlib) [ 331.446809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.528384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.559061] LNet: 9723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.585427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.650029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.677867] LNet: 9730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 338.423786] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:02:44 (1787137364) [ 338.536965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.572494] LNet: 10306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.614653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.681324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 344.370380] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:02:50 (1787137370) [ 344.448645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.466135] LNet: 10887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.488890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.582684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 349.740473] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:02:56 (1787137376) [ 349.810218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.849418] LNet: 11472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.873114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.941564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 355.949434] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:03:02 (1787137382) [ 356.086067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.119211] LNet: 12065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.153830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.228418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 356.645576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.002423] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:03:08 (1787137388) [ 362.159335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.212377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.299937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.417762] hrtimer: interrupt took 3752289 ns [ 362.843453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.460880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 369.359988] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:03:15 (1787137395) [ 369.597197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.625676] LNet: 13253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.636746] LNet: 13253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.662265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.761690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 375.616440] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:03:21 (1787137401) [ 375.718805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.766294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.894538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 375.937695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.380113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 376.440634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.859809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 376.887866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.490516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 377.543788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.993920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 378.059903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.507704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 378.551758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.030607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 379.071565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.757498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 379.788715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 380.218954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 380.242253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 380.689200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 380.729249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 381.310344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 381.373200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 381.752307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 381.790961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 382.130344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 382.167856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.909048] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:03:34 (1787137414) [ 388.009091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.069955] LNet: 14513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.076633] LNet: 14513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 388.097416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.169570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 388.197369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 394.706532] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:03:40 (1787137420) [ 394.870600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.965045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.138219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 395.184955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 405.554382] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:03:51 (1787137431) [ 405.890232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.961541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.044970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 406.076552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 413.825551] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:03:59 (1787137439) [ 413.965773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.017544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.108620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.127610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.453746] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:04:07 (1787137447) [ 421.575758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.610216] LNet: 16860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.619174] LNet: 16860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 421.660600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.818933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.844954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.539452] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:04:14 (1787137454) [ 428.801826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.845318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.924436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.956436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 436.136858] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:04:21 (1787137461) [ 436.253144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.312064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.471535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.115396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.986736] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:04:28 (1787137468) [ 442.151183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.195416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.456787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.662346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 449.398203] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:04:35 (1787137475) [ 449.605256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.653807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.767516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.110238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 457.101739] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:04:42 (1787137482) [ 457.537424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.591727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.299785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.733770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 466.088142] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:04:51 (1787137491) [ 466.223233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.334611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.421583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.462551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.906572] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:04:58 (1787137498) [ 473.053320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.089962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.159334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.191637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.556554] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:05:05 (1787137505) [ 479.713543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.745177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.821971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 485.836871] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:05:11 (1787137511) [ 486.025151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.075270] LNet: 22306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.086780] LNet: 22306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 486.111680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.163703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 491.534736] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:05:17 (1787137517) [ 491.639238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.682727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.774349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 492.134446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 498.584216] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:05:24 (1787137524) [ 498.866444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.921785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.009826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 499.046614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 505.663448] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:05:31 (1787137531) [ 506.064733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.112720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.224978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 506.250661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 511.892965] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:05:38 (1787137538) [ 512.117564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.147850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.225802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 512.276599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 518.859188] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:05:44 (1787137544) [ 519.182105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.207800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.246686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 519.272092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.518175] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:05:51 (1787137551) [ 525.601918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.667230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.709726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 525.751046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 525.761731] LNetError: 25837:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 525.787040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 525.829158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 531.513380] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:05:57 (1787137557) [ 531.882742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.933642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.013323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.elfpe9BbSP [ 532.162914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.elfpe9BbSP [ 532.200847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.elfpe9BbSP [ 538.973930] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:06:04 (1787137564) [ 539.127426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.176410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.252062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 539.302734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 539.365258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 545.973349] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:06:12 (1787137572) [ 546.138888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.245201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.340334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 546.356623] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 547.943791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 549.441867] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 551.431195] Key type lgssc registered [ 552.802481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.066486] LNetError: 27598:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.51@tcp99 has no available nets [ 559.137758] Key type lgssc unregistered [ 559.508594] LNetError: 28707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.550595] LNet: Removed LNI 192.168.201.51@tcp [ 560.305866] Key type .llcrypt unregistered [ 560.311947] Key type ._llcrypt unregistered [ 566.420589] Key type ._llcrypt registered [ 566.422278] Key type .llcrypt registered [ 567.668301] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:06:33 (1787137593) [ 568.065212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.075478] alg: No test for adler32 (adler32-zlib) [ 568.965755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.021194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 572.054021] LNet: Message forwarding will be enabled in 2s [ 575.135897] LNet: Message forwarding enabled [ 575.544090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 575.558528] LNet: Message forwarding will stop in 2s to 6s [ 580.063931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 580.862584] LNet: Message forwarding disabled [ 582.471921] LNet: Message forwarding will be enabled in 2s [ 586.016135] LNet: Message forwarding enabled [ 589.505290] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:06:55 (1787137615) [ 589.643579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.683501] LNet: 30111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.266845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.396309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 597.688312] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:07:03 (1787137623) [ 597.893467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.915233] LNet: 30695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 597.945928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.008973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 598.051678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 598.080674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 598.125906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 598.157461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 598.185161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 598.210509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 598.236423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 598.263313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 598.287485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 598.317740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 598.341912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 598.381973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 598.424091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 598.481496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 598.525939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 598.578370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 598.641943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 598.719076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 598.744496] LNetError: 30738:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 604.518615] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:07:10 (1787137630) [ 604.842926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.867504] LNet: 31309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.891743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.975758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.888532] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:07:18 (1787137638) [ 613.361661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg151-server [ 614.932753] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:07:20 (1787137640) [ 615.074542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.115597] LNet: 32236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.131878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.199838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 615.242302] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 615.945834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[52-56]@tcp [ 615.965880] LNet: 32373:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 616.019298] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 616.048941] LNetError: 32243:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.52@tcp has gone from up to down [ 616.995516] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 617.007977] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 618.024741] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 618.038713] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 620.069695] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.54@tcp is being used as a gateway but routing feature is not turned on [ 620.081843] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 622.551654] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:07:28 (1787137648) [ 624.175823] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 624.205953] LNetError: 32243:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 624.712312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg151-server [ 626.513385] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:07:32 (1787137652) [ 626.733072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.768129] LNet: 33307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.784439] LNetError: 33307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.816827] LNet: Removed LNI 192.168.201.51@tcp [ 627.869619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.921572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.956266] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 628.957622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 629.000983] LNet: 33426:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 629.022968] LNet: 33426:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 629.069889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 629.085402] LNetError: 33313:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.54@tcp has gone from up to down [ 629.099257] LNetError: 33313:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 635.871697] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:07:41 (1787137661) [ 636.076254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.121624] LNet: 34016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.133513] LNetError: 34016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.168073] LNet: Removed LNI 192.168.201.51@tcp [ 637.190781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.299940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 637.359960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.420460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.549824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.634925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.669627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.744444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 637.841468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.867272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.930710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.040166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.091803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.164356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 638.243763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.274644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.330672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.392461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.425505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.501390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 638.584119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.611878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.690456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.787266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.847663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.929075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 639.025839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.064424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.135797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 639.219025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.257976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.351436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 639.463923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.524502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.657204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.947939] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:07:51 (1787137671) [ 647.819447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg151-server [ 649.677555] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:07:55 (1787137675) [ 651.460680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg151-server [ 652.888650] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:07:59 (1787137679) [ 652.956717] LNet: 35412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.962544] LNet: 35412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 653.385558] Key type .llcrypt unregistered [ 653.387030] Key type ._llcrypt unregistered [ 659.132776] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 659.273111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 659.277077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 659.279397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 661.844679] Key type ._llcrypt registered [ 661.847340] Key type .llcrypt registered [ 662.081100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.106534] alg: No test for adler32 (adler32-zlib) [ 663.188348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.268146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.287811] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 663.964518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 663.998698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 666.381486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 666.406831] LNet: Removed LNI 10.1.2.3@tcp [ 666.604719] LNet: 36144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.615791] LNetError: 36144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.643862] LNet: Removed LNI 192.168.201.51@tcp [ 667.620342] Key type .llcrypt unregistered [ 667.622274] Key type ._llcrypt unregistered [ 682.318474] Key type ._llcrypt registered [ 682.321990] Key type .llcrypt registered [ 683.806700] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:08:29 (1787137709) [ 684.290622] Key type .llcrypt unregistered [ 684.292952] Key type ._llcrypt unregistered [ 690.235168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 690.408288] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 690.419063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 690.430691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 693.275846] Key type ._llcrypt registered [ 693.277505] Key type .llcrypt registered [ 693.493768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.510658] alg: No test for adler32 (adler32-zlib) [ 694.480941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.539668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.570906] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 695.362350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 695.374636] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.942771] LNet: 37644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.955905] LNetError: 37644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 697.966247] LNet: Removed LNI 10.1.2.3@tcp [ 697.980610] LNet: Removed LNI 192.168.201.51@tcp [ 698.748420] Key type .llcrypt unregistered [ 698.753147] Key type ._llcrypt unregistered [ 711.764789] Key type ._llcrypt registered [ 711.766394] Key type .llcrypt registered [ 713.541847] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:08:58 (1787137738) [ 714.207464] Key type .llcrypt unregistered [ 714.212430] Key type ._llcrypt unregistered [ 721.037659] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 721.216087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 721.225056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 721.233164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.799397] Key type ._llcrypt registered [ 723.803104] Key type .llcrypt registered [ 723.945866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.973908] alg: No test for adler32 (adler32-zlib) [ 725.011659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.087147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 725.113581] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 726.750268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.769210] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.013941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.036285] LNet: Removed LNI 10.1.2.3@tcp [ 729.045377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 729.075782] LNet: Removed LNI 10.1.2.31@tcp [ 729.107936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.154529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.942782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 730.003939] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 730.867820] LNet: 39574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.872237] LNetError: 39574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 730.876310] LNet: Removed LNI 10.1.2.31@tcp [ 730.887940] LNet: Removed LNI 10.1.2.3@tcp [ 731.469373] Key type .llcrypt unregistered [ 731.472606] Key type ._llcrypt unregistered [ 744.413304] Key type ._llcrypt registered [ 744.415430] Key type .llcrypt registered [ 745.931088] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:09:31 (1787137771) [ 746.349507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 746.359551] alg: No test for adler32 (adler32-zlib) [ 747.296819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.361473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.389413] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 748.425385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.439744] LNet: 40531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.445736] LNetError: 40531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.455360] LNet: Removed LNI 192.168.201.51@tcp [ 748.469601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.536868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.51@tcp [ 748.568565] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 749.494154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.51@tcp [ 749.519740] LNet: Removed LNI 192.168.201.51@tcp [ 749.578943] LNet: 40667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.356462] Key type .llcrypt unregistered [ 750.359279] Key type ._llcrypt unregistered [ 762.982512] Key type ._llcrypt registered [ 762.986221] Key type .llcrypt registered [ 764.236629] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:09:50 (1787137790) [ 766.251235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg151-server [ 768.454985] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:09:54 (1787137794) [ 769.092691] Key type .llcrypt unregistered [ 769.095055] Key type ._llcrypt unregistered [ 775.644122] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 775.820837] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 775.824940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 775.833297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.328493] Key type ._llcrypt registered [ 778.333832] Key type .llcrypt registered [ 778.480967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.500320] alg: No test for adler32 (adler32-zlib) [ 779.358184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.392815] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 779.890714] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 781.705666] LNet: 42507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.713741] LNetError: 42507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.721467] LNet: Removed LNI 192.168.201.51@tcp1 [ 781.732531] LNet: Removed LNI 10.1.2.3@tcp2 [ 782.367473] Key type .llcrypt unregistered [ 782.370050] Key type ._llcrypt unregistered [ 794.305863] Key type ._llcrypt registered [ 794.307757] Key type .llcrypt registered [ 795.578722] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:10:21 (1787137821) [ 796.137406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 796.154389] alg: No test for adler32 (adler32-zlib) [ 797.097902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.158621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.180493] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 798.031333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.087304] LNet: 43482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.105117] LNetError: 43482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.194714] LNet: Removed LNI 192.168.201.51@tcp [ 798.304938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.427865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 798.492128] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 805.801545] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:10:31 (1787137831) [ 806.138640] LNet: 44073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.145159] LNetError: 44073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.165690] LNet: Removed LNI 192.168.201.51@tcp [ 806.850538] Key type .llcrypt unregistered [ 806.853849] Key type ._llcrypt unregistered [ 813.542276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.654573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.664242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.669400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.533835] Key type ._llcrypt registered [ 816.537310] Key type .llcrypt registered [ 816.841748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 816.880318] alg: No test for adler32 (adler32-zlib) [ 817.913915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.035845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.081066] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 819.004310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 819.025597] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.020628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.040799] LNet: 44821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.050093] LNetError: 44821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 822.056245] LNet: Removed LNI 10.1.2.3@tcp [ 822.063694] LNet: Removed LNI 192.168.201.51@tcp [ 822.078863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.158807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 822.190931] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 822.202800] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.786860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.809724] LNet: 44848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.815895] LNetError: 44848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 822.823595] LNetError: 44848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 822.831360] LNet: Removed LNI 10.1.2.3@tcp [ 822.840986] LNet: Removed LNI 192.168.201.51@tcp [ 822.861263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.916946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 822.940215] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 822.944796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.499376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 823.521958] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 824.624723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.649150] LNet: 45011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.658630] LNetError: 45011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.670102] LNetError: 45011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.679667] LNet: Removed LNI 10.1.2.3@tcp [ 824.681463] LNet: Removed LNI 192.168.201.51@tcp [ 824.689498] LNet: Removed LNI 192.168.201.51@tcp2 [ 824.708419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.780523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.816079] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 824.821906] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.826978] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 834.855744] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:11:00 (1787137860) [ 835.003512] LNet: 45601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.014608] LNetError: 45601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 835.021944] LNetError: 45601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 835.027876] LNet: Removed LNI 10.1.2.3@tcp [ 835.030515] LNet: Removed LNI 192.168.201.51@tcp [ 835.049736] LNet: Removed LNI 192.168.201.51@tcp2 [ 835.747757] Key type .llcrypt unregistered [ 835.751986] Key type ._llcrypt unregistered [ 843.842762] Key type ._llcrypt registered [ 843.845751] Key type .llcrypt registered [ 844.051687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.071371] alg: No test for adler32 (adler32-zlib) [ 845.095129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.188509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.244120] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 846.352156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.377255] LNet: 46009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.384293] LNetError: 46009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.402991] LNet: Removed LNI 192.168.201.51@tcp [ 846.465025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.595900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.623026] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 847.216085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.240641] LNet: 46039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.245973] LNetError: 46039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.258807] LNet: Removed LNI 192.168.201.51@tcp [ 847.281943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.353813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.376154] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 847.856830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.895242] LNet: Removed LNI 192.168.201.51@tcp [ 847.916263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.001761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.032904] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 848.715591] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.903496] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.905787] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.909383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.503294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.551968] LNet: 46147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.565472] LNet: 46147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 851.585049] LNetError: 46147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.600767] LNetError: 46147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.643202] LNet: Removed LNI 192.168.201.51@tcp [ 851.697604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.780452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 851.800459] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 859.999262] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:11:25 (1787137885) [ 860.258107] LNet: 46745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.278695] LNetError: 46745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.297413] LNet: Removed LNI 192.168.201.51@tcp [ 860.930736] Key type .llcrypt unregistered [ 860.932135] Key type ._llcrypt unregistered [ 867.795835] Key type ._llcrypt registered [ 867.798649] Key type .llcrypt registered [ 867.995918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.017716] alg: No test for adler32 (adler32-zlib) [ 868.902293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.952735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.977304] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 869.777438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.798928] LNet: 47147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.806536] LNetError: 47147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.823606] LNet: Removed LNI 192.168.201.51@tcp [ 869.846676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.909969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 869.935070] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 877.733615] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:11:43 (1787137903) [ 878.032383] LNet: 47740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.042284] LNetError: 47740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.060552] LNet: Removed LNI 192.168.201.51@tcp [ 879.038941] Key type .llcrypt unregistered [ 879.045984] Key type ._llcrypt unregistered [ 885.976640] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 886.084470] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 886.091158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 886.106214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 888.539476] Key type ._llcrypt registered [ 888.543410] Key type .llcrypt registered [ 888.746720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.772711] alg: No test for adler32 (adler32-zlib) [ 889.733445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.816360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.866881] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 890.689113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.707416] LNet: 48190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.712629] LNetError: 48190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.724903] LNet: Removed LNI 192.168.201.51@tcp [ 890.739715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.784623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 890.802689] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 897.022828] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:12:02 (1787137922) [ 897.225936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.285141] LNet: 48790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.298126] LNetError: 48790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.310578] LNet: Removed LNI 192.168.201.51@tcp [ 897.350959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.425581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 904.212765] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:12:09 (1787137929) [ 904.299474] LNet: 49366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.851673] Key type .llcrypt unregistered [ 904.854615] Key type ._llcrypt unregistered [ 912.743846] Key type ._llcrypt registered [ 912.749023] Key type .llcrypt registered [ 913.141907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.177667] alg: No test for adler32 (adler32-zlib) [ 914.229284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.289392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.306798] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 915.445165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.474302] LNet: 49777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.480573] LNetError: 49777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.496917] LNet: Removed LNI 192.168.201.51@tcp [ 915.522270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.621441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 915.645185] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 923.288295] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:12:28 (1787137948) [ 923.422478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.450183] LNet: 50363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.460780] LNetError: 50363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.484340] LNet: Removed LNI 192.168.201.51@tcp [ 923.516573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.605768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 929.238466] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:12:35 (1787137955) [ 930.624838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 932.132381] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:12:38 (1787137958) [ 933.679819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 935.003532] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:12:41 (1787137961) [ 936.600487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 938.301723] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:12:44 (1787137964) [ 940.260780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 943.007439] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:12:48 (1787137968) [ 944.969991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 947.617751] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:12:52 (1787137972) [ 947.758815] LNet: 52416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.736697] Key type .llcrypt unregistered [ 948.741220] Key type ._llcrypt unregistered [ 956.889948] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.182371] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.188848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.198818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.839116] Key type ._llcrypt registered [ 959.841767] Key type .llcrypt registered [ 959.971154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.981318] alg: No test for adler32 (adler32-zlib) [ 961.002325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.091349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.115712] LNet: Added LNI 192.168.201.51@tcp [8/512/0/180] [ 961.120610] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 961.122504] LNet: 52744:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 961.281949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.319285] LNet: 52763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.342582] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.345893] LNet: Removed LNI 10.1.2.3@tcp [ 961.352575] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.368492] LNet: Removed LNI 192.168.201.51@tcp [ 961.392642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.587172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.642500] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 961.749475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.782602] LNet: 52787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.785978] LNetError: 52787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.797649] LNet: Removed LNI 192.168.201.51@tcp [ 961.812910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.894873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 961.924779] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 961.933245] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.935587] LNet: 52795:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.036823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.068702] LNet: 52813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.078993] LNetError: 52813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.090402] LNet: Removed LNI 10.1.2.3@tcp [ 962.091954] LNetError: 52813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.112805] LNet: Removed LNI 192.168.201.51@tcp [ 962.160840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.257945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.283910] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 962.293439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.296865] LNet: 52821:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 962.666650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.707504] LNet: 52847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.720688] LNetError: 52847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 962.734379] LNet: Removed LNI 10.1.2.3@tcp [ 962.739821] LNetError: 52847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.762831] LNet: Removed LNI 192.168.201.51@tcp [ 962.831865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.892749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 962.909553] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 962.913348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.077495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.096375] LNet: 52875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.114959] LNetError: 52875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 963.120922] LNet: Removed LNI 10.1.2.3@tcp [ 963.123305] LNetError: 52875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.144840] LNet: Removed LNI 192.168.201.51@tcp [ 963.159707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.206972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 963.227422] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 963.237632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 963.240376] LNet: 52883:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 963.364738] LNet: Removed LNI 10.1.2.3@tcp [ 963.395682] LNet: Removed LNI 192.168.201.51@tcp [ 964.216767] Key type .llcrypt unregistered [ 964.218361] Key type ._llcrypt unregistered [ 975.551232] Key type ._llcrypt registered [ 975.553033] Key type .llcrypt registered [ 976.705893] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:13:22 (1787138002) [ 978.243461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 980.231737] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:13:26 (1787138006) [ 980.733368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 980.744366] alg: No test for adler32 (adler32-zlib) [ 981.823552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.877842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.906356] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 982.890884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.51 [ 982.905140] LNet: 54152:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 982.912725] LNet: Removed LNI 192.168.201.51@tcp [ 990.471981] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:13:36 (1787138016) [ 990.616360] LNet: 54718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.623673] LNetError: 54718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.648222] LNet: Removed LNI 192.168.201.51@tcp [ 991.273462] Key type .llcrypt unregistered [ 991.275294] Key type ._llcrypt unregistered [ 998.357986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.576333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.535465] Key type ._llcrypt registered [ 1001.536904] Key type .llcrypt registered [ 1001.665631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.699492] alg: No test for adler32 (adler32-zlib) [ 1002.958904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.524152] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:13:55 (1787138035) [ 1009.918634] LNet: 55639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.930524] LNetError: 55639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.954477] LNet: Removed LNI 10.1.2.3@tcp [ 1010.821398] Key type .llcrypt unregistered [ 1010.823962] Key type ._llcrypt unregistered [ 1018.272168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.532736] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.538665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.548647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1020.325533] Key type ._llcrypt registered [ 1020.333432] Key type .llcrypt registered [ 1020.577219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.591294] alg: No test for adler32 (adler32-zlib) [ 1021.638239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1028.880624] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:14:14 (1787138054) [ 1030.692921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1032.556350] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:14:18 (1787138058) [ 1032.704083] LNet: 56812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.719079] LNetError: 56812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1032.788964] LNet: Removed LNI 10.1.2.3@tcp [ 1033.698843] Key type .llcrypt unregistered [ 1033.701373] Key type ._llcrypt unregistered [ 1040.494275] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1040.811477] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1040.815320] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1040.820670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1043.641294] Key type ._llcrypt registered [ 1043.643684] Key type .llcrypt registered [ 1043.811597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.823605] alg: No test for adler32 (adler32-zlib) [ 1044.813416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.879524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1053.630943] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:14:39 (1787138079) [ 1054.070933] LNet: 58054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.077644] LNetError: 58054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.099254] LNet: Removed LNI 10.1.2.3@tcp [ 1054.810497] Key type .llcrypt unregistered [ 1054.814982] Key type ._llcrypt unregistered [ 1061.693808] Key type ._llcrypt registered [ 1061.696881] Key type .llcrypt registered [ 1061.929348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.954786] alg: No test for adler32 (adler32-zlib) [ 1062.933682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.978033] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1071.478802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1072.067500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1072.111926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1072.178526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1072.210197] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1072.223414] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1072.476713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.511366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.585628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.626724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.663232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.705784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.737388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.788050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.801691] LNet: 58495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.818467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.863684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.928823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.983503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.010503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.036525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.065943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.553312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1073.585668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1073.619887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1073.641659] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1073.937712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.972720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1074.005688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.031303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.057645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.080859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.104789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.149452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.207866] LNet: 58605:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.272239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.319863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.354619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.400954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.438599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.487441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.526287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1075.028154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1075.064234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1075.096053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1075.118511] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1075.128684] LNet: Skipped 1 previous similar message [ 1075.336926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.352748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.368427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.383555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.400761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.420937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.446100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.478443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.500447] LNet: 58714:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.530537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.556803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.594475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.643133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.676380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.715358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.745319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.301630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1076.346842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1076.393211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1076.625367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.650990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.670587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.700900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.719026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.734521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.750599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.769816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.794838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.820759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.851189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.881809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.912845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.946133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.977553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1077.528612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1077.609756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1077.655104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1077.688402] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1077.697773] LNet: Skipped 1 previous similar message [ 1077.983639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.018609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1078.060843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.107824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.156091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.206581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.245137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.274272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.284677] LNet: 58932:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.290282] LNet: 58932:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1078.319917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.389204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.426460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.495882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.540053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.591843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.645451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.280800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1079.315993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1079.360739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1079.684626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.754444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1079.795899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.827989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.867378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.903826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.934836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.966227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.006446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.039595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.067556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1080.114805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1080.146446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.174880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1080.205785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1080.519331] LNet: 59086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.526829] LNetError: 59086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.551566] LNet: Removed LNI 192.168.201.51@tcp [ 1081.093686] Key type .llcrypt unregistered [ 1081.096399] Key type ._llcrypt unregistered [ 1098.327182] Key type ._llcrypt registered [ 1098.328804] Key type .llcrypt registered [ 1099.489623] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:15:25 (1787138125) [ 1100.615514] Key type .llcrypt unregistered [ 1100.617592] Key type ._llcrypt unregistered [ 1107.654852] Key type ._llcrypt registered [ 1107.657393] Key type .llcrypt registered [ 1107.946193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.960396] alg: No test for adler32 (adler32-zlib) [ 1108.844538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1108.868578] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1117.173349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1128.849645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.904177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1128.923623] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1130.415763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1130.440968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1130.472917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1130.502536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1130.538898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1130.586840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1130.618643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1130.656310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1130.700784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1130.722061] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1130.727452] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1130.984605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.016791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.068962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.109962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.134149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.167799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.194707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.217890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.234374] LNet: 60558:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.251685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.301531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.368685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.409661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.456186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.517302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.586518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.228107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1132.323744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1132.358557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1132.392504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1132.422517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1132.454231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1132.484808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1132.524647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1132.576425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1132.627628] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1132.637665] LNet: Skipped 2 previous similar messages [ 1132.948400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.981775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.033395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.082510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.125530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.172180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.224974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.264057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.283029] LNet: 60679:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.299875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.332457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.377677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.411757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.458506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.496693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.542812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.172689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1134.215142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1134.248490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1134.288484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1134.325925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1134.371630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1134.417821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1134.456782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1134.503868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1134.534631] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1134.543535] LNet: Skipped 2 previous similar messages [ 1134.750097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.784857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.816276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.850570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.874985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.915095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.941910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.974541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.998481] LNet: 60801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.029487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.084686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.123664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.169115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.198836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.241681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.266166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1135.896791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1135.950878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1136.005129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1136.069190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1136.121101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1136.179479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1136.238971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1136.290740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1136.345342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1136.618781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.647547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.666688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.680750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.705763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.731321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.761863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.789519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.820816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.847289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.870515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.893563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.929814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.998636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.056965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1137.570625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1137.609163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1137.651423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1137.686585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1137.728995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1137.801380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1137.857458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1137.921705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1138.050350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1138.103974] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1138.119058] LNet: Skipped 5 previous similar messages [ 1138.401338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.432439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.460790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.500051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.525597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.548899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.570698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.600783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.624871] LNet: 61043:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.630908] LNet: 61043:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1138.649843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.677847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.703345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.745341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.782596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.805868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.849768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.392238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1139.418368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1139.449620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1139.486886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1139.523623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1139.541553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1139.558593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1139.600348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1139.626990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1139.924700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.966615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.010685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.048905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.091602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.123869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.158548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.185922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.213970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.247544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.288451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.334610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.381060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.437830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.464685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.306173] LNet: 61244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.316581] LNetError: 61244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.325256] LNet: Removed LNI 192.168.201.51@tcp [ 1147.385728] LNet: Removed LNI 192.168.201.51@tcp1 [ 1147.817980] Key type .llcrypt unregistered [ 1147.827131] Key type ._llcrypt unregistered [ 1164.704735] Key type ._llcrypt registered [ 1164.707323] Key type .llcrypt registered [ 1166.034107] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:16:31 (1787138191) [ 1166.919767] Key type .llcrypt unregistered [ 1166.921240] Key type ._llcrypt unregistered [ 1172.661300] Key type ._llcrypt registered [ 1172.665841] Key type .llcrypt registered [ 1172.958582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1172.993234] alg: No test for adler32 (adler32-zlib) [ 1173.950194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1174.001818] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1181.509918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1182.296933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1182.355278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1182.411961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1182.447287] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1182.664962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.694320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.728504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.797702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.848475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.871527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.932781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.970072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.990847] LNet: 62500:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.011405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.047260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.079456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.109478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.137748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.171525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.216496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.771830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1183.800857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1183.836818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1183.855663] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1184.182238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.231971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.280198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.330758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.369933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.404282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.426415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.456432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.478953] LNet: 62609:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.503274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.549533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.591949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.641736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.687264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.730818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.769236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.336704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1185.367166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1185.397683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1185.430197] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1185.665462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.723616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.755550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.808651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.844077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.867659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.885415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.920373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.943063] LNet: 62718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.970524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.020566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.081403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.123243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.159609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.216196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.262660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1186.568662] LNet: 62763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.574985] LNetError: 62763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.591617] LNet: Removed LNI 192.168.201.51@tcp [ 1187.050246] Key type .llcrypt unregistered [ 1187.059399] Key type ._llcrypt unregistered [ 1205.557570] Key type ._llcrypt registered [ 1205.558914] Key type .llcrypt registered [ 1206.775802] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:17:12 (1787138232) [ 1207.595486] Key type .llcrypt unregistered [ 1207.598357] Key type ._llcrypt unregistered [ 1214.839820] Key type ._llcrypt registered [ 1214.841434] Key type .llcrypt registered [ 1215.081752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.113745] alg: No test for adler32 (adler32-zlib) [ 1216.130343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.185623] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1224.248996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1233.386580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.461069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1233.491879] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1235.066629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1235.100849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1235.133546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1235.185348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1235.282800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1235.341433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1235.418113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1235.510964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1235.558347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1235.592178] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1235.594957] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1235.812870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.847756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.889694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.916784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.946450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.988228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.017744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.075964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.106364] LNet: 64235:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.138965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.191908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.248596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.295430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.347203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.407272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.444298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.479395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.129848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1237.179983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1237.231917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1237.306043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1237.365164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1237.449418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1237.481569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1237.506994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1237.542228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1237.557109] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1237.560406] LNet: Skipped 2 previous similar messages [ 1237.799627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.832277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.868826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.902995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.939798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.967943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.999591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.039813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.091287] LNet: 64359:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.145649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.187785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.231339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.288974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.355761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.409979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.447343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.480333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.069446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1239.106812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1239.139110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1239.182158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1239.232849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1239.276385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1239.317828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1239.356366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1239.384371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1239.401743] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1239.631468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.661461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.702755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.735973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.774725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.815967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.888224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.959572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.995666] LNet: 64482:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.010441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.043621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.068667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.100265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.135569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.152860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.173374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1240.202874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.416346] LNet: 64564:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.423489] LNetError: 64564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.430589] LNet: Removed LNI 192.168.201.51@tcp [ 1245.443529] LNet: Removed LNI 192.168.201.51@tcp1 [ 1245.977946] Key type .llcrypt unregistered [ 1245.980300] Key type ._llcrypt unregistered [ 1263.264971] Key type ._llcrypt registered [ 1263.267215] Key type .llcrypt registered [ 1264.538151] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:18:10 (1787138290) [ 1266.165753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1268.506660] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:18:13 (1787138293) [ 1269.737642] Key type .llcrypt unregistered [ 1269.740812] Key type ._llcrypt unregistered [ 1281.166379] Key type ._llcrypt registered [ 1281.174588] Key type .llcrypt registered [ 1281.500519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.530861] alg: No test for adler32 (adler32-zlib) [ 1282.417391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1282.450695] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1292.636702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1293.282331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1293.324304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1293.353856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1293.371755] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1293.572869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1293.602063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1293.631709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.652880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.677713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.699663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.723944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.735174] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1293.779336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.817728] LNet: 66113:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.846198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.871644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.892905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.918950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.945939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.968342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.307853] LNet: 66156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.313418] LNetError: 66156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.324304] LNet: Removed LNI 192.168.201.51@tcp [ 1294.862990] Key type .llcrypt unregistered [ 1294.864626] Key type ._llcrypt unregistered [ 1317.141355] Key type ._llcrypt registered [ 1317.146126] Key type .llcrypt registered [ 1317.519753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1317.575252] alg: No test for adler32 (adler32-zlib) [ 1318.539945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1318.591589] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1326.652572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1336.548648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.627612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.661935] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1338.267164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1338.324836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1338.389443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1338.440489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1338.502795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1338.550664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1338.634590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1338.705158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1338.748280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1338.780369] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1338.992482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1339.019188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1339.043309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1339.068367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1339.108115] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1339.113655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.113841] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1339.166641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.201319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.221865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.232656] LNet: 67042:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.243704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.270923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.301576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.338386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.372549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.418774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.587065] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1343.601874] LNet: Skipped 2 previous similar messages [ 1344.047542] LNet: 67120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.060131] LNetError: 67120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.078563] LNet: Removed LNI 192.168.201.51@tcp [ 1344.109940] LNet: Removed LNI 192.168.201.51@tcp1 [ 1344.556660] Key type .llcrypt unregistered [ 1344.562359] Key type ._llcrypt unregistered [ 1361.025194] Key type ._llcrypt registered [ 1361.028847] Key type .llcrypt registered [ 1362.495693] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:19:48 (1787138388) [ 1362.895663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1362.909225] alg: No test for adler32 (adler32-zlib) [ 1363.846824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.899318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1363.947323] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1364.917500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1364.938331] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1365.694541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1365.800858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1365.865650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1365.904948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1365.948139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1366.013239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1366.081164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1366.120410] LNetError: 68220:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1366.943583] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1366.943772] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1368.993928] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1369.724864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1372.063402] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1373.176723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1373.208764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1373.263519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.282980] LNet: 68327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.292845] LNetError: 68327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.303280] LNet: Removed LNI 192.168.201.51@tcp [ 1374.307920] LNetError: 68327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.330234] LNet: Removed LNI 192.168.201.51@tcp1 [ 1374.376147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.446655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.474838] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1375.360660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.387504] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1376.335528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1376.387676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.439909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1376.489914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1376.524105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1376.564478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1376.611507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.656508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1376.681893] LNetError: 68600:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1377.508900] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1380.235899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1383.655490] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1383.667645] LNet: Skipped 1 previous similar message [ 1388.398503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1388.471232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1388.501455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1395.847514] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:20:21 (1787138421) [ 1396.054462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.080177] LNet: 69317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.084777] LNetError: 69317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.089303] LNet: Removed LNI 192.168.201.51@tcp [ 1396.101274] LNet: Removed LNI 192.168.201.51@tcp1 [ 1396.112803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.149560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.170695] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1397.032491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.055277] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1397.685936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1397.728214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1397.754377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1397.782147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1397.810430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1397.837946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1397.862598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1397.956552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.177293] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.51@tcp [ 1398.177782] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1398.188166] LNet: Skipped 1 previous similar message [ 1413.516687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.671746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1413.725326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.766995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 500 [ 1413.905239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1420.711232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1420.770271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1420.833607] LNet: 69809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.840588] LNetError: 69809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.848425] LNetError: 69809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1420.860765] LNet: Removed LNI 192.168.201.51@tcp [ 1420.882230] LNet: Removed LNI 192.168.201.51@tcp1 [ 1420.924867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.989376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.031593] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1421.922598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1421.942786] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1422.971548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1423.006500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.069127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1423.129535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1423.181610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1423.241421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1423.298593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1423.353753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1424.031566] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1424.040366] LNet: Skipped 3 previous similar messages [ 1424.847458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.325474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.349835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1435.384376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1441.619859] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:21:07 (1787138467) [ 1442.143520] LNet: 70856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.147368] LNetError: 70856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.152402] LNetError: 70856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1442.155541] LNet: Removed LNI 192.168.201.51@tcp [ 1442.168456] LNet: Removed LNI 192.168.201.51@tcp1 [ 1442.978690] Key type .llcrypt unregistered [ 1442.980348] Key type ._llcrypt unregistered [ 1450.076906] Key type ._llcrypt registered [ 1450.078380] Key type .llcrypt registered [ 1450.327474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.348352] alg: No test for adler32 (adler32-zlib) [ 1451.317771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1451.366055] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1461.978956] Lustre: DEBUG MARKER: Initial discovery [ 1462.282287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1464.866642] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1464.981344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1465.027353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1465.067853] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1465.122969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1465.178317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -l 3 [ 1465.220305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -e local_error [ 1465.251228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m PUT -l 6 [ 1465.760902] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1466.452311] Lustre: DEBUG MARKER: Force oleg151-client.virtnet to discover 192.168.201.151@tcp (in background) [ 1466.526531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1468.111071] Lustre: DEBUG MARKER: Force oleg151-server to discover 192.168.201.51@tcp [ 1469.603755] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1471.171509] Lustre: DEBUG MARKER: Wait for 71616 [ 1473.131060] Lustre: DEBUG MARKER: Finished wait on 71616 [ 1473.214188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1473.262863] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1473.268613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1473.700419] LNet: 72081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.709978] LNetError: 72081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.733144] LNet: Removed LNI 192.168.201.51@tcp [ 1474.246550] Key type .llcrypt unregistered [ 1474.265866] Key type ._llcrypt unregistered [ 1494.511266] Key type ._llcrypt registered [ 1494.513486] Key type .llcrypt registered [ 1495.854901] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:22:01 (1787138521) [ 1496.557353] Key type .llcrypt unregistered [ 1496.561456] Key type ._llcrypt unregistered [ 1503.865253] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1504.112607] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1504.123668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1504.125965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1506.759443] Key type ._llcrypt registered [ 1506.761667] Key type .llcrypt registered [ 1506.974506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1506.991617] alg: No test for adler32 (adler32-zlib) [ 1507.912142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.974889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.006814] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1508.663969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1508.690582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1517.624113] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:22:23 (1787138543) [ 1517.705110] LNet: 74156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.711411] LNetError: 74156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1517.714521] LNet: Removed LNI 10.1.2.3@tcp [ 1517.740314] LNet: Removed LNI 192.168.201.51@tcp [ 1518.436381] Key type .llcrypt unregistered [ 1518.438570] Key type ._llcrypt unregistered [ 1526.091909] Key type ._llcrypt registered [ 1526.094904] Key type .llcrypt registered [ 1526.287839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1526.300636] alg: No test for adler32 (adler32-zlib) [ 1527.207980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.280722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.305224] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1528.012269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1528.028237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1528.965137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.51@tcp [ 1530.015228] LNetError: Unexpected error -22 connecting to 192.168.201.51@tcp at host 192.168.201.51:988 [ 1536.068281] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:22:41 (1787138561) [ 1536.209752] LNet: 75295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.219524] LNetError: 75295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1537.247337] LNet: Removed LNI 10.1.2.3@tcp [ 1537.249883] LNetError: 75295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.287937] LNet: Removed LNI 192.168.201.51@tcp [ 1538.120531] Key type .llcrypt unregistered [ 1538.122188] Key type ._llcrypt unregistered [ 1545.961211] Key type ._llcrypt registered [ 1545.964652] Key type .llcrypt registered [ 1546.250895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1546.274467] alg: No test for adler32 (adler32-zlib) [ 1547.258760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.319821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.343569] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1548.317191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1548.350832] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 1549.381066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.51@tcp1 --nid 192.168.201.51@tcp2 [ 1554.632717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp2 [ 1554.664609] LNetError: 76150:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.51@tcp1 to 192.168.201.51@tcp2 - Route table may be misconfigured [ 1554.689848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp2 192.168.201.51@tcp1 [ 1562.176483] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:23:07 (1787138587) [ 1562.336085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.365497] LNet: 76720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.379181] LNetError: 76720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.386205] LNet: Removed LNI 192.168.201.51@tcp1 [ 1563.442403] LNet: Removed LNI 192.168.201.51@tcp2 [ 1563.495951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.589663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.649625] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1564.574918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.614174] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1565.278973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1565.317905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.51@tcp [ 1565.349110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp1 [ 1565.399698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp -e network_timeout [ 1565.433687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -e network_timeout [ 1565.465099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -e network_timeout [ 1565.498561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -e network_timeout [ 1565.529611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1565.543466] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1565.560316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1565.599222] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1565.632404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.743026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1572.029689] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:23:17 (1787138597) [ 1572.283910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.348909] LNet: 77584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.356390] LNetError: 77584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.360754] LNetError: 77584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1572.372714] LNet: Removed LNI 192.168.201.51@tcp [ 1573.425878] LNet: Removed LNI 192.168.201.51@tcp1 [ 1573.471562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.582526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1573.938834] LNet: 77626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.334518] Key type .llcrypt unregistered [ 1574.337797] Key type ._llcrypt unregistered [ 1586.832716] Key type ._llcrypt registered [ 1586.836220] Key type .llcrypt registered [ 1588.591667] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:23:33 (1787138613) [ 1589.116796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.129788] alg: No test for adler32 (adler32-zlib) [ 1590.114926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.698958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1593.436107] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:23:39 (1787138619) [ 1593.674727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.693927] LNet: 78726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.710971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.766529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.793629] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1594.748626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1594.778373] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1595.566204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1595.599242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp1 [ 1595.633877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp1 [ 1602.450071] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:23:48 (1787138628) [ 1604.289270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg151-server [ 1606.176225] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:23:52 (1787138632) [ 1608.042889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg151-server [ 1609.689363] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:23:55 (1787138635) [ 1611.481898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg151-server [ 1613.181609] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:23:59 (1787138639) [ 1615.212740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg151-server [ 1616.806466] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:24:02 (1787138642) [ 1618.541622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg151-server [ 1619.822313] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:24:06 (1787138646) [ 1621.421423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg151-server [ 1622.927454] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:24:09 (1787138649) [ 1624.549527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg151-server [ 1626.372101] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:24:12 (1787138652) [ 1628.846412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg151-server [ 1630.814803] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:24:16 (1787138656) [ 1632.872926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg151-server [ 1634.673343] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:24:20 (1787138660) [ 1634.947462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.974410] LNet: 82603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.985864] LNetError: 82603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.999325] LNet: Removed LNI 192.168.201.51@tcp [ 1636.001351] LNetError: 82603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.037803] LNet: Removed LNI 192.168.201.51@tcp1 [ 1636.087696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.199652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.226950] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1637.237420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1637.387043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1637.558373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.589771] LNet: 82751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.597722] LNetError: 82751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.617613] LNet: Removed LNI 192.168.201.51@tcp [ 1637.636563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.672654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1637.690473] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1638.484511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1638.669744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.717279] LNet: Removed LNI 192.168.201.51@tcp [ 1639.774198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.873601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.904056] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1640.604980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1640.672428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1640.835866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.860772] LNet: 83043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.867754] LNet: 83043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1640.874516] LNetError: 83043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.881789] LNetError: 83043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1640.898481] LNet: Removed LNI 192.168.201.51@tcp [ 1640.915321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.964964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1640.990926] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1642.011709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1642.269406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.303830] LNet: Removed LNI 192.168.201.51@tcp [ 1643.327698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.431192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.478753] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1644.135850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1644.230699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1644.545640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.613586] LNet: Removed LNI 192.168.201.51@tcp [ 1645.649914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.721607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1645.757301] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1646.683679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1646.921270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.947611] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.952966] LNet: 83481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1646.961810] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.968478] LNetError: 83481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1646.982193] LNet: Removed LNI 192.168.201.51@tcp [ 1647.001500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.084954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.120981] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1648.115917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1648.184890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1648.513733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.583625] LNet: Removed LNI 192.168.201.51@tcp [ 1649.632110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.763897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1649.779794] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1650.753353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1652.211574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.288981] LNet: Removed LNI 192.168.201.51@tcp [ 1653.336543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.434800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.462772] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1654.404974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1654.562102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1654.911940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.949605] LNet: 83931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.955703] LNet: 83931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1654.973925] LNetError: 83931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.988072] LNetError: 83931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1655.033081] LNet: Removed LNI 192.168.201.51@tcp [ 1655.076162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.209147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1655.273538] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1656.414263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1656.670722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.706935] LNet: Removed LNI 192.168.201.51@tcp [ 1657.793811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.897714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.946708] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1658.970694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1659.111915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1659.483607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.523579] LNet: Removed LNI 192.168.201.51@tcp [ 1660.596119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.679993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1660.702853] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1661.797398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1662.205495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.271661] LNet: Removed LNI 192.168.201.51@tcp [ 1663.308903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.415587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1663.462840] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1664.225835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1664.308916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.350994] LNet: Removed LNI 192.168.201.51@tcp [ 1664.402163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.475221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.511620] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1665.389681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1665.464630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.505090] LNet: Removed LNI 192.168.201.51@tcp [ 1665.524791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.574899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1665.654433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.703939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.759581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1665.893512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.938113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.023424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1666.119474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.171226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.249383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1666.292537] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1667.271426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.288918] LNet: Removed LNI 192.168.201.51@tcp [ 1667.297888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.355361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1674.770130] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:25:00 (1787138700) [ 1675.241658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.278865] LNet: 85392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.283778] LNet: 85392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1675.323823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.428432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1675.486311] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1676.740346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.757537] LNetError: 85530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.763381] LNetError: 85530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1676.778792] LNet: Removed LNI 192.168.201.51@tcp231 [ 1676.821732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.900489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1676.926688] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1676.932095] LNet: 85538:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1677.462750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1677.549299] LNet: Removed LNI 192.168.201.51@tcp231 [ 1677.580541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1677.606764] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1678.882963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.923384] LNet: Removed LNI 192.168.201.51@tcp231 [ 1678.952950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.043935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1679.095745] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1679.109407] LNet: 85691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.850867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1679.906363] LNet: Removed LNI 192.168.201.51@tcp231 [ 1679.942740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1679.988895] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1681.893662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.943815] LNet: Removed LNI 192.168.201.51@tcp231 [ 1681.967900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.065065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1682.088403] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1682.091779] LNet: 85842:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1682.541138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1682.569474] LNet: Removed LNI 192.168.201.51@tcp231 [ 1682.584523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1682.616753] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1683.763620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.790970] LNet: Removed LNI 192.168.201.51@tcp231 [ 1683.806821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.879100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1683.913608] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1684.258178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1684.280876] LNet: Removed LNI 192.168.201.51@tcp231 [ 1684.294254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1684.328243] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1692.691599] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:25:18 (1787138718) [ 1692.925614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.984449] LNet: Removed LNI 192.168.201.51@tcp231 [ 1693.009794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.075086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1693.097077] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1694.027469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1694.055788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1694.098124] LNet: Removed LNI 192.168.201.51@tcp232 [ 1694.114427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1694.136986] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1695.107485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1695.212258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1695.258565] LNet: Removed LNI 192.168.201.51@tcp232 [ 1695.296686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1695.370634] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1696.451414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1696.519381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1696.569052] LNet: Removed LNI 192.168.201.51@tcp232 [ 1696.597635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1696.619551] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1697.912795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1697.957311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1697.995257] LNet: Removed LNI 192.168.201.51@tcp232 [ 1698.025664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1706.291909] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:25:31 (1787138731) [ 1707.779700] Key type .llcrypt unregistered [ 1707.782083] Key type ._llcrypt unregistered [ 1717.117598] Key type ._llcrypt registered [ 1717.122378] Key type .llcrypt registered [ 1717.417710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1717.445221] alg: No test for adler32 (adler32-zlib) [ 1718.583520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1718.662873] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1728.450185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1740.255739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.297803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1740.327698] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1742.193846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1742.261942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1742.321983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1742.381922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1742.452577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1742.519536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1742.562524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1742.597466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1742.650465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1742.679872] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1743.127502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.179773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1743.261135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.298578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1743.341863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1743.394459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1743.458800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1743.512728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1743.545589] LNet: 88502:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1743.585660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1743.633927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1743.717713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1743.811860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1743.881311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1743.931830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1744.676754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1744.721244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1744.767544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1744.855375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1744.920276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1744.994371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1745.056413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1745.133443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1745.203519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1745.229162] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1745.233779] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1745.512133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1745.540845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1745.567922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1745.593752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1745.626754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1745.655831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1745.685697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1745.721767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1745.738461] LNet: 88623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.755428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1745.782216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1745.814433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1745.855243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1745.897927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1745.933565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1746.552122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1746.629102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1746.681258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1746.738240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1746.800585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1746.855333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1746.916940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1746.952763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1746.989395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1747.035757] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1747.445916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1747.487483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1747.578467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1747.630759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1747.676532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1747.758735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1747.842441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1747.887478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1747.903179] LNet: 88743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1747.931868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1747.989835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.040487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.080863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.124655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.163426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1748.820646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1748.861153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1748.911530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1748.948265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1749.006485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1749.056091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1749.107464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1749.159676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1749.253605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1749.294641] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1749.615878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1749.665664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1749.693671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1749.743685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1749.811333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1749.875730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1749.933711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1749.980693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1750.002988] LNet: 88864:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.047300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1750.157456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1750.218142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1750.252371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1750.291263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1750.331946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1750.973268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1751.024489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1751.062252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1751.115737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1751.170457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1751.251543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1751.323177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1751.380116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1751.452532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1751.845011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1751.887590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1751.947488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1751.984842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1752.016738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1752.057431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1752.104589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1752.137259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1752.172993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1752.202198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1752.241929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1752.271705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1752.314085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1752.354459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1752.846659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1752.884868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1752.921472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1752.963858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1753.018702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1753.055269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1753.090619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1753.126287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1753.167569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1753.189334] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1753.442963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1753.481492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1753.515402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1753.553584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1753.588858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1753.638666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1753.675432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1753.715849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1753.742662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1753.781840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1753.809186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1753.855251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1753.924265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1753.979814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1760.316774] LNet: 89183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.323220] LNetError: 89183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.375475] LNet: Removed LNI 192.168.201.51@tcp [ 1761.380254] LNetError: 89183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.407509] LNet: Removed LNI 192.168.201.51@tcp1 [ 1762.000488] Key type .llcrypt unregistered [ 1762.004589] Key type ._llcrypt unregistered [ 1781.598936] Key type ._llcrypt registered [ 1781.600942] Key type .llcrypt registered [ 1783.429442] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:26:48 (1787138808) [ 1784.022657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1784.057898] alg: No test for adler32 (adler32-zlib) [ 1785.073753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.124307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1785.148271] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1786.235654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1786.294721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1786.338516] LNetError: 90164:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1786.502782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.51@tcp [ 1786.647744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.004926] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:26:59 (1787138819) [ 1794.152253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.200143] LNet: 90754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.211876] LNetError: 90754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1794.223924] LNet: Removed LNI 192.168.201.51@tcp [ 1794.253743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.302800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.323993] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1794.850595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1794.874229] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1795.796244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.51@tcp [ 1795.820977] LNetError: 91020:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1796.319505] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1796.321391] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1797.091021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.51@tcp [ 1805.944593] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:27:11 (1787138831) [ 1807.793856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg151-server [ 1809.360945] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:27:15 (1787138835) [ 1811.390175] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg151-server [ 1813.085656] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:27:19 (1787138839) [ 1814.807669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg151-server [ 1816.679526] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:27:22 (1787138842) [ 1816.790632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.805612] LNet: 92646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.814649] LNetError: 92646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.823161] LNet: Removed LNI 192.168.201.51@tcp [ 1817.828459] LNetError: 92646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.861708] LNet: Removed LNI 192.168.201.51@tcp1 [ 1817.923129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.984812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.003547] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1818.725073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1818.778706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1818.814398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1818.871799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1818.911144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1818.939289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1826.515087] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:27:32 (1787138852) [ 1826.642089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.711521] LNet: 93370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.720922] LNetError: 93370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.747905] LNet: Removed LNI 192.168.201.51@tcp [ 1826.793887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.936288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1826.973188] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1834.540228] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:27:40 (1787138860) [ 1834.978736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.027655] LNet: 94082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.036789] LNetError: 94082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.053389] LNet: Removed LNI 192.168.201.51@tcp [ 1835.106321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.244487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.266229] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1842.333528] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:27:48 (1787138868) [ 1843.832099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1845.647163] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:27:51 (1787138871) [ 1846.020415] LNet: 95117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.023836] LNetError: 95117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.047297] LNet: Removed LNI 192.168.201.51@tcp [ 1846.778881] Key type .llcrypt unregistered [ 1846.783742] Key type ._llcrypt unregistered [ 1853.130294] Key type ._llcrypt registered [ 1853.132640] Key type .llcrypt registered [ 1853.417305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1853.466890] alg: No test for adler32 (adler32-zlib) [ 1854.426181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1854.479385] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1863.480235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1870.244778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.151@tcp [ 1871.753174] LNet: 95544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.763275] LNetError: 95544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.787773] LNet: Removed LNI 192.168.201.51@tcp [ 1872.230781] Key type .llcrypt unregistered [ 1872.237182] Key type ._llcrypt unregistered [ 1884.881481] Key type ._llcrypt registered [ 1884.883051] Key type .llcrypt registered [ 1886.418672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1888.004804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1890.019417] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:28:35 (1787138915) [ 1891.755531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1893.564296] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:28:39 (1787138919) [ 1895.243464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg151-server [ 1896.567253] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:28:42 (1787138922) [ 1898.861638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg151-server [ 1900.786246] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:28:46 (1787138926) [ 1903.070187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1903.310995] alg: No test for adler32 (adler32-zlib) [ 1904.585120] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 1905.011795] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1906.809632] Key type lgssc registered [ 1908.619798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.094744] Key type lgssc unregistered [ 1933.511779] LNet: 98724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.531098] LNetError: 98724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.570049] LNet: Removed LNI 192.168.201.51@tcp [ 1934.890263] Key type .llcrypt unregistered [ 1934.891962] Key type ._llcrypt unregistered [ 1952.716629] Key type ._llcrypt registered [ 1952.718328] Key type .llcrypt registered [ 1954.369133] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:29:39 (1787138979) [ 1954.830720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.865588] alg: No test for adler32 (adler32-zlib) [ 1955.828231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.896773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.943384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.989941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.047426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1963.119535] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:29:49 (1787138989) [ 1963.358245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.398024] LNet: 100139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.424679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.482215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.522748] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1964.362985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.377585] LNet: 100283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.381939] LNetError: 100283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.395925] LNet: Removed LNI 192.168.201.51@tcp [ 1964.407648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.454945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1964.477520] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1965.129476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.177607] LNetError: 100311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.195862] LNet: Removed LNI 192.168.201.51@tcp [ 1965.227313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.295473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1965.355359] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1974.173775] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:29:59 (1787138999) [ 1974.382570] LNet: 100901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.397548] LNet: 100901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1974.414825] LNetError: 100901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.442161] LNet: Removed LNI 192.168.201.51@tcp [ 1975.125778] Key type .llcrypt unregistered [ 1975.128075] Key type ._llcrypt unregistered [ 1983.862782] Key type ._llcrypt registered [ 1983.865046] Key type .llcrypt registered [ 1984.189377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.203679] alg: No test for adler32 (adler32-zlib) [ 1985.258717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.345752] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1985.347674] LNet: *** cfs_fail_loc=e003, val=100*** [ 1985.961255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1985.971710] LNet: *** cfs_fail_loc=e003, val=100*** [ 1985.973548] LNet: Skipped 2 previous similar messages [ 1985.978094] Ping reply received with mapping: [ 1985.980711] found metadata reply NI[192.168.201.51@tcp] [ 1985.983343] 33 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1985.985468] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1985.987240] 5a 5a 5a 5a 5a 5a 5a 5a [ 1986.167501] LNet: 101303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1986.172263] LNetError: 101303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.186192] LNet: Removed LNI 192.168.201.51@tcp [ 1986.956618] Key type .llcrypt unregistered [ 1986.961967] Key type ._llcrypt unregistered [ 1999.950565] Key type ._llcrypt registered [ 1999.955048] Key type .llcrypt registered [ 2001.140915] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:30:27 (1787139027) [ 2001.735327] Key type .llcrypt unregistered [ 2001.737546] Key type ._llcrypt unregistered [ 2010.351171] Key type ._llcrypt registered [ 2010.370600] Key type .llcrypt registered [ 2010.744674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.763466] alg: No test for adler32 (adler32-zlib) [ 2012.218820] LNetError: 102363:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2012.965824] Key type .llcrypt unregistered [ 2012.970749] Key type ._llcrypt unregistered [ 2018.849973] Key type ._llcrypt registered [ 2018.852904] Key type .llcrypt registered [ 2020.401560] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:30:46 (1787139046) [ 2021.022381] Key type .llcrypt unregistered [ 2021.024762] Key type ._llcrypt unregistered [ 2030.249261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2030.256343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2031.903718] Key type ._llcrypt registered [ 2031.905932] Key type .llcrypt registered [ 2032.291957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2032.328285] alg: No test for adler32 (adler32-zlib) [ 2033.392365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.492595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2045.440070] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:31:10 (1787139070) [ 2045.555740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.594793] LNet: 104135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.603308] LNetError: 104135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2045.643564] LNet: Removed LNI 10.1.2.3@tcp [ 2045.724906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.931711] Key type .llcrypt unregistered [ 2046.934383] Key type ._llcrypt unregistered [ 2054.211507] Key type ._llcrypt registered [ 2054.216351] Key type .llcrypt registered [ 2054.476725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2054.504080] alg: No test for adler32 (adler32-zlib) [ 2055.533047] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2055.597430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2055.699769] LNet: 104436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.703977] LNetError: 104436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.741882] LNet: Removed LNI 192.168.201.51@tcp [ 2057.638765] Key type .llcrypt unregistered [ 2057.644792] Key type ._llcrypt unregistered [ 2070.092463] Key type ._llcrypt registered [ 2070.096083] Key type .llcrypt registered [ 2072.069363] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:31:37 (1787139097) [ 2073.078300] Key type .llcrypt unregistered [ 2073.080399] Key type ._llcrypt unregistered [ 2080.283620] Key type ._llcrypt registered [ 2080.293775] Key type .llcrypt registered [ 2080.502222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2080.512734] alg: No test for adler32 (adler32-zlib) [ 2089.403966] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:31:55 (1787139115) [ 2089.681054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.753079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.888799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2089.924143] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2091.051294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2091.100642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2091.155823] LNet: Removed LNI 192.168.201.51@tcp [ 2091.634324] LNet: 106193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.319838] Key type .llcrypt unregistered [ 2092.327448] Key type ._llcrypt unregistered [ 2106.888304] Key type ._llcrypt registered [ 2106.889772] Key type .llcrypt registered [ 2108.762927] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:32:14 (1787139134) [ 2109.347974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2109.378407] alg: No test for adler32 (adler32-zlib) [ 2110.340223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.379181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2110.414706] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2111.559350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2111.655727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.51@tcp [ 2111.692444] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2111.704736] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2111.712499] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2111, 0, 0 [ 2111.722884] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2111.728871] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2111.739576] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2111.751192] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2111, 0, 0 [ 2111.766619] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2111.774789] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2111.780225] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2111.788497] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2111, 0, 2161 [ 2111.790712] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2111.799794] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2111.812752] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2111.818468] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2111, 0, 2161 [ 2111.823085] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2119.503535] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:32:25 (1787139145) [ 2120.166397] LNet: 107733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.175793] LNetError: 107733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.191472] LNet: Removed LNI 192.168.201.51@tcp [ 2121.217545] Key type .llcrypt unregistered [ 2121.220728] Key type ._llcrypt unregistered [ 2129.331481] Key type ._llcrypt registered [ 2129.334393] Key type .llcrypt registered [ 2129.534620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2129.556880] alg: No test for adler32 (adler32-zlib) [ 2130.501761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2130.539149] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2139.346567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2150.789193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.856134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2150.892639] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2152.036517] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2152.196909] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2152.209878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2152.219155] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2154.611141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2154.635871] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2167.482417] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:33:12 (1787139192) [ 2167.657480] LNet: 109240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2167.666761] LNetError: 109240:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.673137] LNet: Removed LNI 192.168.201.51@tcp [ 2167.684129] LNet: Removed LNI 192.168.201.51@tcp1 [ 2167.709616] LNet: Removed LNI 10.1.2.3@tcp99 [ 2168.610642] Key type .llcrypt unregistered [ 2168.612432] Key type ._llcrypt unregistered [ 2176.730359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2176.746696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2179.612447] Key type ._llcrypt registered [ 2179.623691] Key type .llcrypt registered [ 2179.909809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2179.948388] alg: No test for adler32 (adler32-zlib) [ 2181.084653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.175184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.220775] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2181.991680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2182.014512] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.997849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2185.095330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2185.206758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2185.250531] LNetError: 109995:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2185.312417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2185.412948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2186.033746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2186.169382] LNet: 110015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.176608] LNetError: 110015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2186.186558] LNet: Removed LNI 10.1.2.3@tcp [ 2187.237349] LNet: Removed LNI 192.168.201.51@tcp [ 2188.342562] Key type .llcrypt unregistered [ 2188.345684] Key type ._llcrypt unregistered [ 2202.470350] Key type ._llcrypt registered [ 2202.476288] Key type .llcrypt registered [ 2204.293750] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:33:49 (1787139229) [ 2204.762868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.777944] alg: No test for adler32 (adler32-zlib) [ 2205.798422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.923983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.993035] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2207.032416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2207.124485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg151-client@tcp [ 2214.763722] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:34:00 (1787139240) [ 2214.896911] LNet: 111546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.901710] LNetError: 111546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.923381] LNet: Removed LNI 192.168.201.51@tcp [ 2215.808549] Key type .llcrypt unregistered [ 2215.810174] Key type ._llcrypt unregistered [ 2223.226881] Key type ._llcrypt registered [ 2223.229243] Key type .llcrypt registered [ 2223.391665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.405168] alg: No test for adler32 (adler32-zlib) [ 2224.414384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2224.438660] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2224.489941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2231.815789] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:34:17 (1787139257) [ 2233.784325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2235.782509] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:34:21 (1787139261) [ 2238.494622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2241.262453] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:34:26 (1787139266) [ 2241.376947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.393745] LNet: 112987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.399122] LNetError: 112987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2241.418640] LNet: Removed LNI 192.168.201.51@tcp [ 2241.466901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.558432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.618555] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2242.786557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2242.891985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2252.442779] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:34:37 (1787139277) [ 2252.624871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.683431] LNet: 113695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.709970] LNetError: 113695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.739834] LNet: Removed LNI 192.168.201.51@tcp [ 2252.762598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.801807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2252.847482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2253.412569] Key type .llcrypt unregistered [ 2253.414254] Key type ._llcrypt unregistered [ 2268.667672] Key type ._llcrypt registered [ 2268.671861] Key type .llcrypt registered [ 2270.117773] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:34:55 (1787139295) [ 2270.677339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2270.694252] alg: No test for adler32 (adler32-zlib) [ 2271.760640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.804444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.836947] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2272.826749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2272.888592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2280.258715] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:35:06 (1787139306) [ 2280.394606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.416217] LNet: 115256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2280.427108] LNetError: 115256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.450407] LNet: Removed LNI 192.168.201.51@tcp [ 2280.491084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.593937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2280.635966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2286.607673] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:35:12 (1787139312) [ 2286.880895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.906289] LNet: 115842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.931131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.019445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2287.418184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2287.448580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2287.544097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2287.886937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2287.917961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2288.002848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.037826] LNet: 115875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.069350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.120325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2288.686804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2288.721816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2288.790141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2289.274266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2289.294791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2296.343971] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:35:22 (1787139322) [ 2296.461394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.514782] LNet: 116469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.581829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.644723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.666522] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2297.408431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2297.430430] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2298.548942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2298.614323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2298.956555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.987879] LNetError: 116754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.000894] LNet: Removed LNI 192.168.201.51@tcp [ 2299.017867] LNet: Removed LNI 192.168.201.51@tcp2 [ 2299.125944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.188271] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2299.194908] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2299.199391] LNet: 116760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2299.618737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2299.899620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.959562] LNet: Removed LNI 192.168.201.51@tcp [ 2300.962604] LNetError: 116798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.980633] LNetError: 116798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2301.002442] LNet: Removed LNI 192.168.201.51@tcp2 [ 2301.034640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.071344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.090836] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2302.019402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2302.040050] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2303.029691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2303.095561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2303.207389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2303.422538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.440703] LNet: 117082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.444832] LNet: 117082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2303.450523] LNetError: 117082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.455788] LNet: Removed LNI 192.168.201.51@tcp [ 2304.485822] LNet: Removed LNI 192.168.201.51@tcp2 [ 2304.502488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.605546] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2304.616236] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2304.628512] LNet: 117088:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2305.305825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2305.503286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.520399] LNet: Removed LNI 192.168.201.51@tcp [ 2306.540848] LNet: Removed LNI 192.168.201.51@tcp2 [ 2306.601963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.727689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2306.776378] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2307.749843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2307.771086] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2309.121987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2309.228716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2309.621283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.672052] LNetError: 117405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.682980] LNetError: 117405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2309.692716] LNet: Removed LNI 192.168.201.51@tcp [ 2309.710027] LNet: Removed LNI 192.168.201.51@tcp2 [ 2309.764427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.920077] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2309.945304] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2309.962308] LNet: 117411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2311.189509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2311.564703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.599415] LNet: 117448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.606340] LNet: 117448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2312.671989] LNet: Removed LNI 192.168.201.51@tcp [ 2312.685059] LNet: Removed LNI 192.168.201.51@tcp2 [ 2312.754061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.850716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.892199] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2313.758783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2313.780960] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2314.807469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2314.842087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2314.887343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2315.032471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.069059] LNet: Removed LNI 192.168.201.51@tcp [ 2317.095420] LNet: Removed LNI 192.168.201.51@tcp2 [ 2317.156452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.233885] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2317.241894] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2317.246557] LNet: 117739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2318.131305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2318.507524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.534398] LNetError: 117776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.539755] LNetError: 117776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2318.548224] LNet: Removed LNI 192.168.201.51@tcp [ 2319.590075] LNet: Removed LNI 192.168.201.51@tcp2 [ 2319.630144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.698932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.739568] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2320.584265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2320.609506] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2321.836506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2321.893809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2322.006675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2322.632984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.679166] LNet: Removed LNI 192.168.201.51@tcp [ 2323.696550] LNet: Removed LNI 192.168.201.51@tcp2 [ 2323.736934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.827449] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2323.835939] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2323.842342] LNet: 118079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.552945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2325.201476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.303323] LNet: Removed LNI 192.168.201.51@tcp [ 2326.321796] LNet: Removed LNI 192.168.201.51@tcp2 [ 2326.400494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.480787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.517215] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2327.359352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2327.384649] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2328.589066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2328.654449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2328.729938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2328.818567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2329.303252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.334580] LNet: 118433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.345890] LNet: 118433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2330.399361] LNet: Removed LNI 192.168.201.51@tcp [ 2331.429889] LNet: Removed LNI 192.168.201.51@tcp2 [ 2331.475874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.597130] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2331.607423] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2332.464529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2340.610381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2342.452172] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:36:08 (1787139368) [ 2342.940098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.976732] LNetError: 119199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2342.990624] LNetError: 119199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2343.003126] LNet: Removed LNI 192.168.201.51@tcp [ 2344.037817] LNet: Removed LNI 192.168.201.51@tcp2 [ 2344.065596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.499721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.528048] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2345.375224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2345.413756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2346.277517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.303619] LNet: Removed LNI 10.1.2.3@tcp [ 2346.319285] LNet: Removed LNI 192.168.201.51@tcp [ 2346.340168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.416710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2346.444985] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2346.449052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2346.926520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2346.965701] LNet: Removed LNI 192.168.201.51@tcp [ 2346.991320] LNet: Removed LNI 10.1.2.3@tcp [ 2354.139551] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:36:19 (1787139379) [ 2354.276578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.362504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.457887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.491767] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2355.243630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2355.287957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.329501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.363184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.397254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.431508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.463487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.501948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.549666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.580398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.651856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2355.816632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2362.580475] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:36:28 (1787139388) [ 2362.874872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.903881] LNet: 120813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.908548] LNet: 120813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2363.945070] LNet: Removed LNI 192.168.201.51@tcp [ 2363.998468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.193499] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2364.310780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2364.321569] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2364.334947] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2366.759909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2366.795941] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2367.576203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2367.604263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2369.348508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2369.380921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2369.430597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.51@tcp -m PUT -e local_timeout -r 1 [ 2369.493593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2369.544734] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2369.556110] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2369.852746] LNetError: 120879:0:(lib-move.c:4801:lnet_parse()) 192.168.201.51@tcp, src 192.168.201.51@tcp: Dropping PUT (error -108 looking up sender) [ 2370.922493] LNet: Removed LNI 10.1.2.3@tcp [ 2372.010101] LNet: Removed LNI 192.168.201.51@tcp [ 2372.443978] Key type .llcrypt unregistered [ 2372.447582] Key type ._llcrypt unregistered [ 2386.393707] Key type ._llcrypt registered [ 2386.395208] Key type .llcrypt registered [ 2387.986814] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:36:53 (1787139413) [ 2388.597874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.609951] alg: No test for adler32 (adler32-zlib) [ 2389.542868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.012328] LNet: 122195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.946860] Key type .llcrypt unregistered [ 2390.948527] Key type ._llcrypt unregistered [ 2399.189542] Key type ._llcrypt registered [ 2399.192077] Key type .llcrypt registered [ 2399.441254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2399.485448] alg: No test for adler32 (adler32-zlib) [ 2400.472605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2400.501906] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2409.719889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2409.967558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2410.589658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2410.807390] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 2410.923529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2410.950238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2410.977418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2410.994504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2411.008158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2411.021712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2411.041273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2411.063655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2411.077909] LNet: 122649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2411.091397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2411.111581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2411.133210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2411.154498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2411.178631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2411.229521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2411.876891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2412.126425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2412.151384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2412.173916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2412.199454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2412.225746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2412.256511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2412.305333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2412.343866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2412.377866] LNet: 122766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2412.404790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2412.443394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2412.481891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2412.504920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2412.520858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2412.539562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2413.247426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2413.582507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2413.642726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2413.690673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2413.746301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2413.806589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2413.862842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2413.911543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2413.958060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2413.996940] LNet: 122883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.029898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.096472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.130367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.155217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.180531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.214760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.035328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2415.518879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2415.562445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2415.595842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2415.620742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2415.641582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2415.680037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2415.725976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2415.765517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2415.795857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2415.815520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2415.846152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2415.878891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2415.914279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2415.972301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2416.462502] LNet: 123051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2416.470685] LNetError: 123051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2416.499778] LNet: Removed LNI 192.168.201.51@tcp [ 2416.895967] Key type .llcrypt unregistered [ 2416.899607] Key type ._llcrypt unregistered [ 2436.447901] Key type ._llcrypt registered [ 2436.451413] Key type .llcrypt registered [ 2438.377712] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:37:43 (1787139463) [ 2438.799819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2438.820452] alg: No test for adler32 (adler32-zlib) [ 2439.783553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.295822] LNet: 123938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.144406] Key type .llcrypt unregistered [ 2441.151493] Key type ._llcrypt unregistered [ 2448.630251] Key type ._llcrypt registered [ 2448.633218] Key type .llcrypt registered [ 2448.811666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2448.825622] alg: No test for adler32 (adler32-zlib) [ 2449.788742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2449.849854] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2458.468675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2469.005112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.059356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2469.111882] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2470.111877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2470.732324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2470.959546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2470.999457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2471.038881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2471.072442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2471.123534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2471.191984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2471.215761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2471.238635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2471.253858] LNet: 124595:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2471.271633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2471.295755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2471.326392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2471.351424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2471.374640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2471.396304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2471.931813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2472.130506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2472.151876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2472.174150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2472.196916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2472.224547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2472.252912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2472.280467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2472.312212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2472.329725] LNet: 124712:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2472.357570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2472.394683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2472.415402] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.151@tcp, 192.168.201.151@tcp1 [ 2472.434249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2472.470281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2472.500266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2472.553183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2473.194814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2473.424944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2473.455514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2473.486609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2473.507528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2473.528972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2473.553254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2473.592626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2473.627146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2473.653526] LNet: 124829:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2473.670521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2473.705448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2473.743403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2473.772614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2473.808283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2473.846404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2474.475233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2474.728856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2474.763233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2474.794951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2474.825817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2474.853401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2474.885281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2474.916830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2474.947119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2474.981184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2475.024540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2475.062395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2475.105545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2475.157444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2475.202761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2480.848718] LNet: 125031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.859838] LNetError: 125031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.873674] LNet: Removed LNI 192.168.201.51@tcp [ 2480.887795] LNet: Removed LNI 192.168.201.51@tcp1 [ 2481.336660] Key type .llcrypt unregistered [ 2481.338494] Key type ._llcrypt unregistered [ 2501.210769] Key type ._llcrypt registered [ 2501.212246] Key type .llcrypt registered [ 2503.100252] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:38:48 (1787139528) [ 2503.705480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.722737] alg: No test for adler32 (adler32-zlib) [ 2504.663337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2504.762387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2504.822419] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2505.867876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2505.934025] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2506.957465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2507.086139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2508.127146] LNet: Removed LNI 192.168.201.51@tcp [ 2515.756822] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:39:01 (1787139541) [ 2517.665593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2519.570530] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:39:05 (1787139545) [ 2528.366549] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:39:14 (1787139554) [ 2535.971985] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:39:21 (1787139561) [ 2544.091334] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:39:29 (1787139569) [ 2551.494220] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:39:37 (1787139577) [ 2560.803951] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:39:46 (1787139586) [ 2567.089526] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:39:53 (1787139593) [ 2574.070198] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:39:59 (1787139599) [ 2582.003438] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:40:07 (1787139607) [ 2589.741823] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:40:15 (1787139615) [ 2598.401251] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:40:23 (1787139623) [ 2607.132934] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:40:32 (1787139632) [ 2615.339630] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:40:40 (1787139640) [ 2623.105352] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:40:49 (1787139649) [ 2631.044612] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:40:56 (1787139656) [ 2640.635868] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:41:06 (1787139666) [ 2649.166883] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:41:15 (1787139675) [ 2656.527242] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:41:22 (1787139682) [ 2656.735323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2656.774102] LNet: 136733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2656.782682] LNetError: 136733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2656.811972] LNet: Removed LNI 192.168.201.51@tcp2 [ 2656.865586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2656.979750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2657.002354] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2658.614711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2658.647223] LNet: 136881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2658.657136] LNetError: 136881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2658.685499] LNet: Removed LNI 192.168.201.51@tcp [ 2658.712343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2658.781844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2658.819242] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2666.225967] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:41:32 (1787139692) [ 2673.431311] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:41:39 (1787139699) [ 2680.163514] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:41:45 (1787139705) [ 2687.726909] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:41:53 (1787139713) [ 2693.359726] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:41:59 (1787139719) [ 2699.707992] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:42:06 (1787139726) [ 2708.494718] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:42:13 (1787139733) [ 2718.358118] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:42:23 (1787139743) [ 2726.812253] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:42:33 (1787139753) [ 2734.856546] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:42:40 (1787139760) [ 2742.741750] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:42:48 (1787139768) [ 2750.002584] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:42:56 (1787139776) [ 2756.991811] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2513 sec ========== 07:43:03 (1787139783) [ 2757.339255] LNet: 144345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.347303] LNetError: 144345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.374471] LNet: Removed LNI 192.168.201.51@tcp [ 2758.166811] Key type .llcrypt unregistered [ 2758.169080] Key type ._llcrypt unregistered [ 2771.642085] Key type ._llcrypt registered [ 2771.645740] Key type .llcrypt registered [ 2772.219772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2772.231161] alg: No test for adler32 (adler32-zlib) [ 2773.354567] Lustre: Lustre: Build Version: 2.17.57_8_gdef02f8 [ 2773.574109] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2775.271182] Key type lgssc registered [ 2776.532468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2844.960917] Lustre: Mounted lustre-client - version 2.17.57_8_gdef02f8 [ 2850.008569] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2859.619528] Lustre: Unmounted lustre-client [ 2884.199895] Key type lgssc unregistered [ 2884.509602] LNet: 147309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2884.516066] LNetError: 147309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2884.534191] LNet: Removed LNI 192.168.201.51@tcp [ 2885.105146] Key type .llcrypt unregistered [ 2885.106197] Key type ._llcrypt unregistered