[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442609055 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003276] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008624] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013032] SELinux: Initializing. [ 0.014045] *** VALIDATE selinux *** [ 0.022268] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026691] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028032] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030127] *** VALIDATE tmpfs *** [ 0.031474] *** VALIDATE proc *** [ 0.032224] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034220] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040740] debug: unmapping init [mem 0xffffffff9f859000-0xffffffff9f860fff] [ 0.043170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045020] ... version: 2 [ 0.045853] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.046755] ... value mask: 0000ffffffffffff [ 0.047010] ... max period: 00007fffffffffff [ 0.048007] ... fixed-purpose events: 3 [ 0.048729] ... event mask: 000000070000000f [ 0.049254] rcu: Hierarchical SRCU implementation. [ 0.051228] smp: Bringing up secondary CPUs ... [ 0.052384] x86: Booting SMP configuration: [ 0.053032] .... node #0, CPUs: #1 #2 #3 [ 0.056009] smp: Brought up 1 node, 4 CPUs [ 0.057948] smpboot: Max logical packages: 1 [ 0.058016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148234] node 0 deferred pages initialised in 89ms [ 0.150643] devtmpfs: initialized [ 0.151230] x86/mm: Memory block size: 128MB [ 0.153688] gcov: version magic: 0x41383552 [ 0.155365] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159058] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160218] pinctrl core: initialized pinctrl subsystem [ 0.161171] [ 0.161705] ************************************************************* [ 0.162012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163015] ** ** [ 0.164016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.166011] ** This means that this kernel is built to expose internal ** [ 0.167024] ** IOMMU data structures, which may compromise security on ** [ 0.168009] ** your system. ** [ 0.169013] ** ** [ 0.170013] ** If you see this message and you are not debugging the ** [ 0.171011] ** kernel, report this immediately to your vendor! ** [ 0.172015] ** ** [ 0.173014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174013] ************************************************************* [ 0.175626] NET: Registered protocol family 16 [ 0.176322] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179453] cpuidle: using governor menu [ 0.180000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.182461] PCI: Using configuration type 1 for base access [ 0.184095] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.194091] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198038] cryptd: max_cpu_qlen set to 1000 [ 0.200233] ACPI: Added _OSI(Module Device) [ 0.201009] ACPI: Added _OSI(Processor Device) [ 0.202016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207570] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213365] ACPI: Interpreter enabled [ 0.214043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215011] ACPI: Using IOAPIC for interrupt routing [ 0.216093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217357] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224654] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226033] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.235323] acpiphp: Slot [2] registered [ 0.237129] acpiphp: Slot [5] registered [ 0.238102] acpiphp: Slot [6] registered [ 0.239118] acpiphp: Slot [3] registered [ 0.241112] acpiphp: Slot [4] registered [ 0.242090] acpiphp: Slot [7] registered [ 0.243115] acpiphp: Slot [8] registered [ 0.245101] acpiphp: Slot [9] registered [ 0.246114] acpiphp: Slot [10] registered [ 0.247108] acpiphp: Slot [11] registered [ 0.249124] acpiphp: Slot [12] registered [ 0.250113] acpiphp: Slot [13] registered [ 0.251109] acpiphp: Slot [14] registered [ 0.253116] acpiphp: Slot [15] registered [ 0.254100] acpiphp: Slot [16] registered [ 0.255089] acpiphp: Slot [17] registered [ 0.257124] acpiphp: Slot [18] registered [ 0.258104] acpiphp: Slot [19] registered [ 0.259086] acpiphp: Slot [20] registered [ 0.261112] acpiphp: Slot [21] registered [ 0.262088] acpiphp: Slot [22] registered [ 0.263090] acpiphp: Slot [23] registered [ 0.265092] acpiphp: Slot [24] registered [ 0.266113] acpiphp: Slot [25] registered [ 0.267131] acpiphp: Slot [26] registered [ 0.269093] acpiphp: Slot [27] registered [ 0.270098] acpiphp: Slot [28] registered [ 0.272104] acpiphp: Slot [29] registered [ 0.273115] acpiphp: Slot [30] registered [ 0.274123] acpiphp: Slot [31] registered [ 0.276062] PCI host bridge to bus 0000:00 [ 0.277013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.285023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.286139] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287641] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.289835] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294363] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.297505] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.304653] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.310054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.312682] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.316010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.325010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334662] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.341018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.347015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.359015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367060] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.374015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.377020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.385016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.400829] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.405418] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.408033] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.413311] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.415204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.418119] iommu: Default domain type: Passthrough [ 0.420611] SCSI subsystem initialized [ 0.421098] ACPI: bus type USB registered [ 0.423103] usbcore: registered new interface driver usbfs [ 0.424089] usbcore: registered new interface driver hub [ 0.426103] usbcore: registered new device driver usb [ 0.428187] pps_core: LinuxPPS API ver. 1 registered [ 0.430011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.432052] PTP clock support registered [ 0.433147] EDAC MC: Ver: 3.0.0 [ 0.434398] PCI: Using ACPI for IRQ routing [ 0.435661] NetLabel: Initializing [ 0.436007] NetLabel: domain hash size = 128 [ 0.437009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.438083] NetLabel: unlabeled traffic allowed by default [ 0.439092] vgaarb: loaded [ 0.440149] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.442009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.446073] clocksource: Switched to clocksource kvm-clock [ 0.540700] VFS: Disk quotas dquot_6.6.0 [ 0.541965] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.543770] *** VALIDATE ramfs *** [ 0.544637] *** VALIDATE hugetlbfs *** [ 0.545704] pnp: PnP ACPI init [ 0.548552] pnp: PnP ACPI: found 6 devices [ 0.562965] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.565961] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.568192] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.570347] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.572426] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.574254] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.576826] NET: Registered protocol family 2 [ 0.580377] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.585392] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.588724] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.593854] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.597363] TCP: Hash tables configured (established 65536 bind 65536) [ 0.600404] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.603078] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.605081] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.607602] NET: Registered protocol family 1 [ 0.610146] RPC: Registered named UNIX socket transport module. [ 0.611911] RPC: Registered udp transport module. [ 0.613819] RPC: Registered tcp transport module. [ 0.615238] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.617147] NET: Registered protocol family 44 [ 0.618733] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.620781] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.623220] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.625654] PCI: CLS 0 bytes, default 64 [ 0.627220] Unpacking initramfs... [ 1.948234] debug: unmapping init [mem 0xffff89c3fcc64000-0xffff89c3fffcffff] [ 1.951996] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.953911] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.956807] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.405226] Initialise system trusted keyrings [ 2.406303] Key type blacklist registered [ 2.407484] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.414627] zbud: loaded [ 2.417408] *** VALIDATE nfs *** [ 2.418493] *** VALIDATE nfs4 *** [ 2.420082] pstore: using deflate compression [ 2.423156] Platform Keyring initialized [ 2.503482] NET: Registered protocol family 38 [ 2.504684] Key type asymmetric registered [ 2.505624] Asymmetric key parser 'x509' registered [ 2.507398] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.510717] io scheduler mq-deadline registered [ 2.512485] io scheduler kyber registered [ 2.514192] io scheduler bfq registered [ 2.515444] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.517238] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.519310] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.521586] ACPI: Power Button [PWRF] [ 2.524760] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.532350] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.540232] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.568473] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.598767] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.602218] Non-volatile memory driver v1.3 [ 2.603216] Linux agpgart interface v0.103 [ 2.628781] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.632049] vda: detected capacity change from 0 to 76791808 [ 2.646210] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.648304] vdb: detected capacity change from 0 to 1073741824 [ 2.652261] libphy: Fixed MDIO Bus: probed [ 2.658995] usbcore: registered new interface driver usbserial_generic [ 2.662187] usbserial: USB Serial support registered for generic [ 2.665454] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.671120] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.673241] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.675831] mousedev: PS/2 mouse device common for all mice [ 2.678483] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.680368] rtc_cmos 00:05: RTC can wake from S4 [ 2.684973] rtc_cmos 00:05: registered as rtc0 [ 2.685403] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.686522] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.692397] intel_pstate: CPU model not supported [ 2.692455] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.697061] hid: raw HID events driver (C) Jiri Kosina [ 2.699680] usbcore: registered new interface driver usbhid [ 2.702138] usbhid: USB HID core driver [ 2.704071] drop_monitor: Initializing network drop monitor service [ 2.706986] Initializing XFRM netlink socket [ 2.709293] NET: Registered protocol family 10 [ 2.712809] Segment Routing with IPv6 [ 2.714471] NET: Registered protocol family 17 [ 2.716537] mpls_gso: MPLS GSO support [ 2.723664] RAS: Correctable Errors collector initialized. [ 2.725876] AVX version of gcm_enc/dec engaged. [ 2.727604] AES CTR mode by8 optimization enabled [ 2.791776] sched_clock: Marking stable (2791745504, 0)->(3574746164, -783000660) [ 2.795299] registered taskstats version 1 [ 2.796572] Loading compiled-in X.509 certificates [ 2.798044] zswap: loaded using pool lzo/zbud [ 2.825344] Key type big_key registered [ 2.836438] Key type encrypted registered [ 2.838421] ima: No TPM chip found, activating TPM-bypass! [ 2.840400] ima: Allocated hash algorithm: sha1 [ 2.842289] ima: No architecture policies found [ 2.844216] evm: Initialising EVM extended attributes: [ 2.846129] evm: security.selinux [ 2.847515] evm: security.ima [ 2.848365] evm: security.capability [ 2.849614] evm: HMAC attrs: 0x1 [ 2.851895] rtc_cmos 00:05: setting system clock to 2026-09-10 13:56:05 UTC (1789048565) [ 2.859068] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 2.862360] debug: unmapping init [mem 0xffffffff9f582000-0xffffffff9f858fff] [ 2.869230] Write protecting the kernel read-only data: 28672k [ 2.872776] debug: unmapping init [mem 0xffffffff9dc03000-0xffffffff9ddfffff] [ 2.875480] debug: unmapping init [mem 0xffffffff9e514000-0xffffffff9e5fffff] [ 2.913657] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.923116] systemd[1]: Detected virtualization kvm. [ 2.924737] systemd[1]: Detected architecture x86-64. [ 2.926302] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.952855] systemd[1]: No hostname configured. [ 2.954885] systemd[1]: Set hostname to . [ 2.957342] random: systemd: uninitialized urandom read (16 bytes read) [ 2.960167] systemd[1]: Initializing machine ID from random generator. [ 3.112566] random: systemd: uninitialized urandom read (16 bytes read) [ 3.116703] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.122310] random: systemd: uninitialized urandom read (16 bytes read) [ 3.130578] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.140101] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.760588] device-mapper: uevent: version 1.0.3 [ 3.763204] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.511827] virtio_net virtio0 ens2: renamed from eth0 [ 4.514729] scsi host0: ata_piix [ 4.516744] scsi host1: ata_piix [ 4.517996] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.520323] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.338308] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.355790] random: crng init done [ 9.357803] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.802363] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.947244] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.201234] SELinux: Disabled at runtime. [ 11.267772] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.277438] systemd[1]: Detected virtualization kvm. [ 11.279706] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.742701] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.746267] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.755081] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.759605] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.763727] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.770751] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.775434] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ 11.842772] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.220178] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.551535] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.600708] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.668403] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.691937] EDAC sbridge: Ver: 1.1.2 [ 13.835910] Key type dns_resolver registered [ 14.141957] NFS: Registering the id_resolver key type [ 14.143906] Key type id_resolver registered [ 14.145429] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg214-client login: [ 61.900106] libcfs: loading out-of-tree module taints kernel. [ 62.297585] Key type ._llcrypt registered [ 62.299182] Key type .llcrypt registered [ 63.147634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 63.171915] alg: No test for adler32 (adler32-zlib) [ 64.794379] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 65.959055] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 67.855535] Key type lgssc registered [ 70.085578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 217.964038] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 223.404392] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.824185] hrtimer: interrupt took 2354741 ns [ 238.598905] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing check_logdir /tmp/testlogs/ [ 243.679180] Lustre: lustre-OST0000-osc-ffff89c458abc800: disconnect after 23s idle [ 245.176525] Lustre: DEBUG MARKER: oleg214-client.virtnet: executing yml_node [ 249.947616] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 252.166132] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 255.340130] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 257.887816] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 10 10:00:18 EDT 2026 [ 277.641743] Lustre: Unmounted lustre-client [ 306.498290] Key type lgssc unregistered [ 306.856497] LNet: 8439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.863870] LNetError: 8439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.885444] LNet: Removed LNI 192.168.202.14@tcp [ 308.195163] Key type .llcrypt unregistered [ 308.198898] Key type ._llcrypt unregistered [ 338.629450] Key type ._llcrypt registered [ 338.631537] Key type .llcrypt registered [ 338.876738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.901540] alg: No test for adler32 (adler32-zlib) [ 339.925378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 340.036034] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 340.072838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 344.750784] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 344.992249] LNet: 9343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.001297] LNetError: 9343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.038033] LNet: Removed LNI 192.168.202.14@tcp [ 345.963724] Key type .llcrypt unregistered [ 345.969203] Key type ._llcrypt unregistered [ 353.524626] Key type ._llcrypt registered [ 353.528408] Key type .llcrypt registered [ 355.027465] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:01:55 (1789048915) [ 355.523706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.536861] alg: No test for adler32 (adler32-zlib) [ 356.462554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.547576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.566919] LNet: 9754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.584325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.636383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.653897] LNet: 9761:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 364.077540] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:02:05 (1789048925) [ 364.334393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.400813] LNet: 10341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.437760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.573924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 372.419823] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:02:13 (1789048933) [ 372.586792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.611731] LNet: 10926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.661886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.734734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 380.366043] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:02:21 (1789048941) [ 380.522298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.550153] LNet: 11507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.584961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.707958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 388.234553] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:02:29 (1789048949) [ 388.431450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.467922] LNet: 12095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.543656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.706110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 389.123361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 396.957824] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:02:38 (1789048958) [ 397.064279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.087300] LNet: 12686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.124576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.219896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 397.644521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 397.994055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 403.481594] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:02:45 (1789048965) [ 403.555147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.579756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.698251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 409.593636] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:02:51 (1789048971) [ 409.711750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.866585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.951657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 409.979557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 410.389734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 410.422759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 411.177651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 411.240281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 411.795276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 411.903651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 412.927701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 413.000494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 413.411250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 413.459271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 413.961098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 414.042779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 414.735304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 414.800573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 415.545744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 415.633503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 416.446908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 416.533401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 417.197520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 417.248566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 417.969510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 418.021642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 418.794036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 418.841707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 428.991123] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:03:10 (1789048990) [ 429.483651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.527369] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.535918] LNet: 14552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 429.575242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.753156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 429.803974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 436.833948] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:03:17 (1789048997) [ 437.096395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.198166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.314541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 437.339791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 444.354896] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:03:25 (1789049005) [ 444.605694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.635235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.699948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 444.728615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 452.328875] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:03:33 (1789049013) [ 452.425642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.485145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.613933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 452.668219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 460.071340] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:03:41 (1789049021) [ 460.230309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.273946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.344655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 460.392771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 468.390074] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:03:49 (1789049029) [ 468.704401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.745219] LNet: 17484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.750852] LNet: 17484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 468.794706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.921101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.968652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 476.918706] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:03:57 (1789049037) [ 477.005779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.053717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.145874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.451246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.282951] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:04:05 (1789049045) [ 484.595145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.672450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.957935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 485.203607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.867805] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:04:13 (1789049053) [ 492.028200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.074936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.188554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.398797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.990620] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:04:20 (1789049060) [ 499.066985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.110990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.806361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.036420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 506.724478] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:04:27 (1789049067) [ 506.836512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.893097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.991566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.051157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.069809] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:04:35 (1789049075) [ 514.261473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.332838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.437749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 514.485605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 521.764463] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:04:43 (1789049083) [ 521.854600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.896769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.047547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 528.927125] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:04:50 (1789049090) [ 529.069371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.116841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.211411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 535.699566] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:04:56 (1789049096) [ 536.178229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.196157] LNet: 22922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.201474] LNet: 22922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 536.243487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.345749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 537.017281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 544.805971] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:05:06 (1789049106) [ 545.035491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.078879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.171600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 545.219104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 552.937255] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:05:13 (1789049113) [ 553.155949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.245152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.415857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 553.447914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 562.009908] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:05:22 (1789049122) [ 562.106448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.190883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.334708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 562.374138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 570.792315] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:05:31 (1789049131) [ 570.941689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.005299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.160718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 571.222655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 579.757183] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:05:40 (1789049140) [ 579.985537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.052887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.126836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 580.185338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 580.201473] LNetError: 25870:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 580.237346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 580.269852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 587.669902] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:05:48 (1789049148) [ 587.773712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.876829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.992848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jS1vSUB3t0 [ 588.119072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jS1vSUB3t0 [ 588.163415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jS1vSUB3t0 [ 595.007711] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:05:56 (1789049156) [ 595.287939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.384872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.476615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 595.530851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 595.579985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 603.134186] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:06:04 (1789049164) [ 603.693512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.734541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.789453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.803393] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 604.960127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 607.039717] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 609.056823] Key type lgssc registered [ 610.647620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 615.882809] LNetError: 27631:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.202.14@tcp99 has no available nets [ 619.371764] Key type lgssc unregistered [ 619.692915] LNetError: 28738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.713504] LNet: Removed LNI 192.168.202.14@tcp [ 620.492614] Key type .llcrypt unregistered [ 620.496693] Key type ._llcrypt unregistered [ 629.680242] Key type ._llcrypt registered [ 629.688329] Key type .llcrypt registered [ 631.090448] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:06:32 (1789049192) [ 631.931294] Key type .llcrypt unregistered [ 631.933679] Key type ._llcrypt unregistered [ 640.117115] Key type ._llcrypt registered [ 640.121888] Key type .llcrypt registered [ 640.410926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.462253] alg: No test for adler32 (adler32-zlib) [ 641.415626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.479475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 641.504113] LNet: 29648:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 641.527556] LNet: 29648:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 641.534885] LNet: 29648:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 641.567489] LNet: Message forwarding will be enabled in 2s [ 644.639329] LNet: Message forwarding enabled [ 645.186820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 645.216945] LNet: Message forwarding will stop in 2s to 6s [ 648.738067] LNet: Message forwarding disabled [ 648.891051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 648.913354] LNet: 29737:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 648.920023] LNet: 29737:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 648.929686] LNet: 29737:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 648.962119] LNet: Message forwarding will be enabled in 2s [ 649.272487] LNet: 29750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.120491] Key type .llcrypt unregistered [ 650.124687] Key type ._llcrypt unregistered [ 663.582812] Key type ._llcrypt registered [ 663.585519] Key type .llcrypt registered [ 665.356441] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:07:06 (1789049226) [ 665.834946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 665.857425] alg: No test for adler32 (adler32-zlib) [ 666.834853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.953783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 673.519130] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:07:14 (1789049234) [ 673.744172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.796450] LNet: 31148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.850752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.932875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 673.974911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 674.021871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 674.070963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 674.118893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 674.168758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 674.226141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 674.297193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 674.359621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 674.406135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 674.474254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 674.536976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 674.592627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 674.617507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 674.673862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 674.722806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 674.772108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 674.821626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 674.870846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 674.919515] LNetError: 31194:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 682.249582] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:07:23 (1789049243) [ 682.438742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.484885] LNet: 31765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.520625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.606299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.400769] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:07:30 (1789049250) [ 691.117955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg214-server [ 692.684802] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:07:34 (1789049254) [ 692.876705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.923542] LNet: 32698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.949835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.001831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 693.021828] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 693.954372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[15-19]@tcp [ 693.973823] LNet: 32831:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 694.064460] LNetError: 32702:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.15@tcp is being used as a gateway but routing feature is not turned on [ 694.086648] LNetError: 32702:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.15@tcp has gone from up to down [ 700.884247] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:07:42 (1789049262) [ 702.956612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg214-server [ 704.692624] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:07:45 (1789049265) [ 704.809362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.831285] LNet: 33760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.836958] LNetError: 33760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.879147] LNet: Removed LNI 192.168.202.14@tcp [ 704.966891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.101748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.159093] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 705.947900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[15-20/2]@tcp [ 706.003575] LNet: 33885:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 706.009783] LNet: 33885:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 706.042706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[15-20/2]@tcp [ 706.052805] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.17@tcp is being used as a gateway but routing feature is not turned on [ 706.073066] LNetError: 33770:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 706.083055] LNetError: 33770:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.17@tcp has gone from up to down [ 706.099525] LNetError: 33770:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 711.906632] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:07:53 (1789049273) [ 712.172784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.206851] LNet: 34477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.214667] LNetError: 34477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.229637] LNet: Removed LNI 192.168.202.14@tcp [ 712.242562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.332621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 712.384841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.410417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.458386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 712.508289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.561299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.665179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 712.801348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.850226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.937634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.007563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.047407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.121682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 713.215738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.264423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.344547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.443325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.507640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.566145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 713.688291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.738310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.798445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.905504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.964151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.041408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.147826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.184708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.315874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 714.451589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.532149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.603706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 714.656833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.697619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.750479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 720.455589] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:08:01 (1789049281) [ 722.177974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg214-server [ 723.965333] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:08:05 (1789049285) [ 725.602781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg214-server [ 727.718518] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:08:08 (1789049288) [ 727.840115] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.847271] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 728.673548] Key type .llcrypt unregistered [ 728.679162] Key type ._llcrypt unregistered [ 735.467227] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.644105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.647175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.653532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 738.275125] Key type ._llcrypt registered [ 738.280547] Key type .llcrypt registered [ 738.672295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.706617] alg: No test for adler32 (adler32-zlib) [ 739.712924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.784953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.837650] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 740.421875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.441027] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.483476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 742.529600] LNet: Removed LNI 10.1.2.3@tcp [ 742.695176] LNet: 36592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.705494] LNetError: 36592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.731126] LNet: Removed LNI 192.168.202.14@tcp [ 743.500958] Key type .llcrypt unregistered [ 743.503982] Key type ._llcrypt unregistered [ 757.875387] Key type ._llcrypt registered [ 757.878242] Key type .llcrypt registered [ 759.700986] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:08:40 (1789049320) [ 760.470507] Key type .llcrypt unregistered [ 760.472986] Key type ._llcrypt unregistered [ 769.301993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.656334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.680621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.704851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.474243] Key type ._llcrypt registered [ 771.477225] Key type .llcrypt registered [ 771.765314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.786703] alg: No test for adler32 (adler32-zlib) [ 772.696395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.883694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.963731] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 773.900667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 773.951390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.468622] LNet: 38074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.488317] LNetError: 38074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 778.495793] LNet: Removed LNI 10.1.2.3@tcp [ 778.521375] LNet: Removed LNI 192.168.202.14@tcp [ 779.619514] Key type .llcrypt unregistered [ 779.627684] Key type ._llcrypt unregistered [ 795.266349] Key type ._llcrypt registered [ 795.267354] Key type .llcrypt registered [ 797.462911] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:09:17 (1789049357) [ 798.266280] Key type .llcrypt unregistered [ 798.270943] Key type ._llcrypt unregistered [ 804.174045] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.424319] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.428535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 804.436058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.276656] Key type ._llcrypt registered [ 807.278897] Key type .llcrypt registered [ 807.442828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 807.463756] alg: No test for adler32 (adler32-zlib) [ 808.603901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.678996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 808.709184] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 811.988840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 812.039990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.468632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 815.505375] LNet: Removed LNI 10.1.2.3@tcp [ 815.518229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 815.559894] LNet: Removed LNI 10.1.2.31@tcp [ 815.599727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.656657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.883930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 816.924475] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 818.082221] LNet: 39990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.087872] LNetError: 39990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 818.095979] LNet: Removed LNI 10.1.2.31@tcp [ 818.117293] LNet: Removed LNI 10.1.2.3@tcp [ 818.785659] Key type .llcrypt unregistered [ 818.789459] Key type ._llcrypt unregistered [ 834.415360] Key type ._llcrypt registered [ 834.418387] Key type .llcrypt registered [ 836.234956] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:09:56 (1789049396) [ 836.677638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.690768] alg: No test for adler32 (adler32-zlib) [ 837.638575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.733790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.759370] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 838.665563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 838.684964] LNet: 40940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.691211] LNetError: 40940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.702066] LNet: Removed LNI 192.168.202.14@tcp [ 838.732344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.866619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.14@tcp [ 838.907672] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 839.770298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.14@tcp [ 839.800671] LNet: Removed LNI 192.168.202.14@tcp [ 839.873293] LNet: 41075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.899487] Key type .llcrypt unregistered [ 840.910303] Key type ._llcrypt unregistered [ 855.766467] Key type ._llcrypt registered [ 855.769154] Key type .llcrypt registered [ 857.020079] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:10:18 (1789049418) [ 859.457454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg214-server [ 861.535134] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:10:22 (1789049422) [ 862.090323] Key type .llcrypt unregistered [ 862.091979] Key type ._llcrypt unregistered [ 870.489769] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 870.767405] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 870.783033] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 870.796422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 873.484498] Key type ._llcrypt registered [ 873.486547] Key type .llcrypt registered [ 873.804616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.833591] alg: No test for adler32 (adler32-zlib) [ 874.861200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.958769] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 875.856936] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 878.322707] LNet: 42907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.330304] LNetError: 42907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.336433] LNet: Removed LNI 192.168.202.14@tcp1 [ 878.352365] LNet: Removed LNI 10.1.2.3@tcp2 [ 879.139599] Key type .llcrypt unregistered [ 879.143381] Key type ._llcrypt unregistered [ 893.095108] Key type ._llcrypt registered [ 893.100086] Key type .llcrypt registered [ 894.488372] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:10:55 (1789049455) [ 895.190473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.212164] alg: No test for adler32 (adler32-zlib) [ 896.094405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.153510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.177324] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 897.126989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.144596] LNet: 43874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.153812] LNetError: 43874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.171852] LNet: Removed LNI 192.168.202.14@tcp [ 897.189925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.240164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 897.266355] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 905.117881] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:11:06 (1789049466) [ 905.327433] LNet: 44465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.336468] LNetError: 44465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.369287] LNet: Removed LNI 192.168.202.14@tcp [ 906.179939] Key type .llcrypt unregistered [ 906.181723] Key type ._llcrypt unregistered [ 913.368479] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 913.650409] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 913.664970] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 913.677982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 916.358910] Key type ._llcrypt registered [ 916.368743] Key type .llcrypt registered [ 916.552271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 916.565865] alg: No test for adler32 (adler32-zlib) [ 917.611588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.702838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 917.732798] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 918.557078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 918.578733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.132686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.165388] LNet: 45208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.172444] LNetError: 45208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.178413] LNet: Removed LNI 10.1.2.3@tcp [ 921.202620] LNet: Removed LNI 192.168.202.14@tcp [ 921.247617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.338931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.373420] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 921.376595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.095530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.124382] LNet: 45235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.162871] LNetError: 45235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 922.171074] LNetError: 45235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 922.181525] LNet: Removed LNI 10.1.2.3@tcp [ 922.191575] LNet: Removed LNI 192.168.202.14@tcp [ 922.218773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.298370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 922.333770] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 922.356159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.021959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 923.055585] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 924.406881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.423159] LNet: 45397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.430508] LNetError: 45397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.433670] LNetError: 45397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.438421] LNet: Removed LNI 10.1.2.3@tcp [ 924.441311] LNet: Removed LNI 192.168.202.14@tcp [ 924.450499] LNet: Removed LNI 192.168.202.14@tcp2 [ 924.489692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.581028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 924.619679] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 924.624890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.633091] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 933.209545] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:11:34 (1789049494) [ 933.380360] LNet: 45985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.408245] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 933.424695] LNetError: 45985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 933.448155] LNet: Removed LNI 10.1.2.3@tcp [ 933.451290] LNet: Removed LNI 192.168.202.14@tcp [ 933.464043] LNet: Removed LNI 192.168.202.14@tcp2 [ 934.341918] Key type .llcrypt unregistered [ 934.345209] Key type ._llcrypt unregistered [ 943.178643] Key type ._llcrypt registered [ 943.183628] Key type .llcrypt registered [ 943.476627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.500437] alg: No test for adler32 (adler32-zlib) [ 944.485928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.577492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.619068] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 945.809450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.832664] LNet: 46387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.847429] LNetError: 46387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.860932] LNet: Removed LNI 192.168.202.14@tcp [ 945.893920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.952698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 945.991830] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 946.577824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.631477] LNet: 46417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.637412] LNetError: 46417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.662071] LNet: Removed LNI 192.168.202.14@tcp [ 946.692197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.800110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 946.829621] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 947.335233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.385963] LNet: Removed LNI 192.168.202.14@tcp [ 947.408447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.465847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 947.521607] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 948.061091] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.365543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.389807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.407712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.803559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 949.833860] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.839276] LNet: 46516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 949.858535] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.871598] LNetError: 46516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 949.900433] LNet: Removed LNI 192.168.202.14@tcp [ 949.950097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.036861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 950.076753] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 957.378350] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:11:58 (1789049518) [ 957.624419] LNet: 47113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.628518] LNetError: 47113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.650739] LNet: Removed LNI 192.168.202.14@tcp [ 958.328070] Key type .llcrypt unregistered [ 958.329481] Key type ._llcrypt unregistered [ 965.718900] Key type ._llcrypt registered [ 965.723799] Key type .llcrypt registered [ 966.049979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.096761] alg: No test for adler32 (adler32-zlib) [ 967.113663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.176338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.212372] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 968.334483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.366159] LNet: 47509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.386142] LNetError: 47509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.402314] LNet: Removed LNI 192.168.202.14@tcp [ 968.417717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.480731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 968.504249] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 977.106205] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:12:18 (1789049538) [ 977.242196] LNet: 48096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.251926] LNetError: 48096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.265202] LNet: Removed LNI 192.168.202.14@tcp [ 978.155966] Key type .llcrypt unregistered [ 978.159293] Key type ._llcrypt unregistered [ 987.652888] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 988.043799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 988.051911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 988.066843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 990.763522] Key type ._llcrypt registered [ 990.765165] Key type .llcrypt registered [ 991.121484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 991.153340] alg: No test for adler32 (adler32-zlib) [ 992.184834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.314581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 992.355973] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 993.592138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.625860] LNet: 48545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.633646] LNetError: 48545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.647140] LNet: Removed LNI 192.168.202.14@tcp [ 993.677231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.741564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 993.762240] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1001.966417] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:12:42 (1789049562) [ 1002.073883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.105285] LNet: 49142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.110847] LNetError: 49142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.138828] LNet: Removed LNI 192.168.202.14@tcp [ 1002.168812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.272473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1009.135129] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:12:50 (1789049570) [ 1009.374199] LNet: 49728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.054886] Key type .llcrypt unregistered [ 1010.068716] Key type ._llcrypt unregistered [ 1017.093853] Key type ._llcrypt registered [ 1017.095747] Key type .llcrypt registered [ 1017.285336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.308865] alg: No test for adler32 (adler32-zlib) [ 1018.238340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.328866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1018.379427] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1019.139480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.149483] LNet: 50122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.154648] LNetError: 50122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.165609] LNet: Removed LNI 192.168.202.14@tcp [ 1019.176141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.219319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1019.240789] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1026.515750] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:13:07 (1789049587) [ 1026.738489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.774777] LNet: 50716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.783755] LNetError: 50716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.807950] LNet: Removed LNI 192.168.202.14@tcp [ 1026.819913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.886868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1033.556573] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:13:14 (1789049594) [ 1035.111251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1036.786284] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:13:18 (1789049598) [ 1038.555666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1040.590899] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:13:21 (1789049601) [ 1042.194859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1044.261740] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:13:25 (1789049605) [ 1046.275147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1048.173393] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:13:29 (1789049609) [ 1049.624902] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1051.377502] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:13:32 (1789049612) [ 1051.682572] LNet: 52738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.221551] Key type .llcrypt unregistered [ 1052.223500] Key type ._llcrypt unregistered [ 1058.254058] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1058.575462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1058.584077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1058.596979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1061.281248] Key type ._llcrypt registered [ 1061.286871] Key type .llcrypt registered [ 1061.565866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.587717] alg: No test for adler32 (adler32-zlib) [ 1062.616321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.796835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1062.851925] LNet: Added LNI 192.168.202.14@tcp [8/512/0/180] [ 1062.874238] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1062.876110] LNet: 53050:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1063.008255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.039753] LNet: 53069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.047954] LNetError: 53069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.057468] LNet: Removed LNI 10.1.2.3@tcp [ 1063.061552] LNetError: 53069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.081984] LNet: Removed LNI 192.168.202.14@tcp [ 1063.118877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.214205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.262036] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1063.375141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.408493] LNet: 53093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.418753] LNetError: 53093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.429204] LNet: Removed LNI 192.168.202.14@tcp [ 1063.452402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.518871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.550201] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1063.555842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.561328] LNet: 53101:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1063.653251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1063.689384] LNet: 53119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.700946] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.711615] LNet: Removed LNI 10.1.2.3@tcp [ 1063.716230] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1063.749216] LNet: Removed LNI 192.168.202.14@tcp [ 1063.772356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.865467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1063.892477] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1063.901890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.908777] LNet: 53127:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1064.215711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.272499] LNet: 53153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.290683] LNetError: 53153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.301305] LNet: Removed LNI 10.1.2.3@tcp [ 1064.312287] LNetError: 53153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.340778] LNet: Removed LNI 192.168.202.14@tcp [ 1064.368368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.475417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.506534] LNet: Added LNI 192.168.202.14@tcp [8/256/0/182] [ 1064.510806] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1064.659176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.683602] LNet: 53181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.692972] LNetError: 53181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.701955] LNet: Removed LNI 10.1.2.3@tcp [ 1064.704824] LNetError: 53181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.725507] LNet: Removed LNI 192.168.202.14@tcp [ 1064.754868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.864546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.886960] LNet: Added LNI 192.168.202.14@tcp [8/256/0/182] [ 1064.891454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1064.893429] LNet: 53189:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1064.970664] LNet: Removed LNI 10.1.2.3@tcp [ 1064.982143] LNet: Removed LNI 192.168.202.14@tcp [ 1065.911069] Key type .llcrypt unregistered [ 1065.915407] Key type ._llcrypt unregistered [ 1079.600612] Key type ._llcrypt registered [ 1079.611849] Key type .llcrypt registered [ 1081.201819] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:14:02 (1789049642) [ 1083.012587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1084.818992] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:14:05 (1789049645) [ 1085.298570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.314508] alg: No test for adler32 (adler32-zlib) [ 1086.208526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.285610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1086.313477] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1087.342217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.14 [ 1087.370218] LNet: 54452:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1087.382050] LNet: Removed LNI 192.168.202.14@tcp [ 1094.801620] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:14:15 (1789049655) [ 1095.048079] LNet: 55019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.055573] LNetError: 55019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.107191] LNet: Removed LNI 192.168.202.14@tcp [ 1095.986884] Key type .llcrypt unregistered [ 1095.990704] Key type ._llcrypt unregistered [ 1103.495180] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1103.727922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1106.486517] Key type ._llcrypt registered [ 1106.489220] Key type .llcrypt registered [ 1106.645936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1106.676217] alg: No test for adler32 (adler32-zlib) [ 1107.941972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1114.932226] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:14:36 (1789049676) [ 1115.087559] LNet: 55928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.093930] LNetError: 55928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1115.110134] LNet: Removed LNI 10.1.2.3@tcp [ 1115.638767] Key type .llcrypt unregistered [ 1115.640932] Key type ._llcrypt unregistered [ 1121.613076] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1121.876120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1121.880179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1121.883188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1124.528431] Key type ._llcrypt registered [ 1124.530227] Key type .llcrypt registered [ 1124.718911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.748738] alg: No test for adler32 (adler32-zlib) [ 1125.676522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1133.800123] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:14:54 (1789049694) [ 1135.586238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1137.667435] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:14:58 (1789049698) [ 1137.855665] LNet: 57111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.873043] LNetError: 57111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1137.921099] LNet: Removed LNI 10.1.2.3@tcp [ 1138.735328] Key type .llcrypt unregistered [ 1138.737305] Key type ._llcrypt unregistered [ 1147.820651] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1148.341129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1148.352722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1148.364967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1151.444949] Key type ._llcrypt registered [ 1151.447261] Key type .llcrypt registered [ 1151.624126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1151.639534] alg: No test for adler32 (adler32-zlib) [ 1152.758273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.908686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1162.593365] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:15:23 (1789049723) [ 1163.213614] LNet: 58339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.232369] LNetError: 58339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.265808] LNet: Removed LNI 10.1.2.3@tcp [ 1164.249507] Key type .llcrypt unregistered [ 1164.251422] Key type ._llcrypt unregistered [ 1171.421551] Key type ._llcrypt registered [ 1171.439231] Key type .llcrypt registered [ 1171.743556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.766845] alg: No test for adler32 (adler32-zlib) [ 1172.694203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.720480] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1183.050381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1183.713341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_interrupt -r 1 [ 1183.818033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_interrupt -r 1 [ 1183.904672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1183.933925] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1184.039964] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1184.210750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.239478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.264778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.283875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.310640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1184.351610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.395616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.422906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.475956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.494994] LNet: 58777:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.521483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.578119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.614663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.651277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.710750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.760661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.803887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.091040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_dropped -r 1 [ 1185.126465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_dropped -r 1 [ 1185.161506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1185.178625] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1185.184772] LNet: Skipped 1 previous similar message [ 1185.402390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.450425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.491450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.518643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.551715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1185.593248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.619319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.643400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.675580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.695258] LNet: 58888:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.708132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.733749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.757763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.777314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.800428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.823350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.844838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.445876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_aborted -r 1 [ 1186.520489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_aborted -r 1 [ 1186.563852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1186.588618] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1186.945557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.986174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.046595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.122518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.173856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1187.235940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.304586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.342543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.369271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.391392] LNet: 58999:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.416724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.440666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.465997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.518416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.552666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.589399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.650117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.458051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_no_route -r 1 [ 1188.488897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_no_route -r 1 [ 1188.531815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1188.749380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.788387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.834525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.893638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.000236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1189.032380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.083184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.135361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.156697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.176623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.194887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.217300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.239546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.258550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.280914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.303989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.926814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_timeout -r 1 [ 1189.961436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_timeout -r 1 [ 1190.004279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1190.031827] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1190.038709] LNet: Skipped 1 previous similar message [ 1190.269713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.284935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.308681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.329338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.349559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1190.377548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.398514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.415661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.434596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.445965] LNet: 59222:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.448454] LNet: 59222:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1190.458242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.485337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.513240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.553407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.578380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.600953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.624964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.112624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_error -r 1 [ 1191.136731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_error -r 1 [ 1191.161671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1191.427793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.465194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.504761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.539748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.573585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.608084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.662777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.699889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.737567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.784769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.822326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.857549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.886217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.954561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.025926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.119897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1192.535255] LNet: 59374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.564611] LNetError: 59374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.599707] LNet: Removed LNI 192.168.202.14@tcp [ 1193.396469] Key type .llcrypt unregistered [ 1193.397926] Key type ._llcrypt unregistered [ 1217.366465] Key type ._llcrypt registered [ 1217.368516] Key type .llcrypt registered [ 1218.774563] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:16:19 (1789049779) [ 1219.673501] Key type .llcrypt unregistered [ 1219.676499] Key type ._llcrypt unregistered [ 1228.321532] Key type ._llcrypt registered [ 1228.326173] Key type .llcrypt registered [ 1228.636330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.680985] alg: No test for adler32 (adler32-zlib) [ 1229.741620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1229.771873] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1239.106290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1252.068085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.149963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1252.193268] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1253.782527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_interrupt -r 1 [ 1253.824582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_interrupt -r 1 [ 1253.880801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_interrupt -r 1 [ 1253.939424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_interrupt -r 1 [ 1253.978689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_interrupt -r 1 [ 1254.033080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_interrupt -r 1 [ 1254.072492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_interrupt -r 1 [ 1254.108366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_interrupt -r 1 [ 1254.137159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1254.146937] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1254.150028] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1254.406918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.442523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.475220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.503454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.541559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1254.583402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.629275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.693565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.718224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.732050] LNet: 60840:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.744926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.768958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.786356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.802864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.825977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.862901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.896089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1255.534458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_dropped -r 1 [ 1255.605508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_dropped -r 1 [ 1255.680355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_dropped -r 1 [ 1255.748740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_dropped -r 1 [ 1255.823154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_dropped -r 1 [ 1255.883467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_dropped -r 1 [ 1255.959158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_dropped -r 1 [ 1256.049883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_dropped -r 1 [ 1256.141345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1256.198544] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1256.206621] LNet: Skipped 2 previous similar messages [ 1256.517284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.549450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.594657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.653653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.709550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.772946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.838481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.888539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.957564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.989249] LNet: 60963:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.017645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.083643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.146738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.178560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.250192] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1257.251870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.264593] LNet: Skipped 3 previous similar messages [ 1257.329148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.368991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.973025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_aborted -r 1 [ 1258.012322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_aborted -r 1 [ 1258.071956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_aborted -r 1 [ 1258.150907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_aborted -r 1 [ 1258.233644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_aborted -r 1 [ 1258.296320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_aborted -r 1 [ 1258.332647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_aborted -r 1 [ 1258.371720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_aborted -r 1 [ 1258.411578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1258.719582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1258.757347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1258.777367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.804274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.875942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.921921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.968739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.012883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.050088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.079907] LNet: 61087:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.103851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.131767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.164971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.202287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.232691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.288853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.336874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.166341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_no_route -r 1 [ 1260.203168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_no_route -r 1 [ 1260.261040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_no_route -r 1 [ 1260.307585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_no_route -r 1 [ 1260.340694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_no_route -r 1 [ 1260.368984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_no_route -r 1 [ 1260.390330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_no_route -r 1 [ 1260.418550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_no_route -r 1 [ 1260.442814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1260.455578] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1260.464665] LNet: Skipped 3 previous similar messages [ 1260.643237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.668500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.701578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.739964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.783263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1260.832356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.895982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.929901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.973536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.017969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.088412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.166467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.217450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.250513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.284753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.319679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.749486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_timeout -r 1 [ 1261.782932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_timeout -r 1 [ 1261.840388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_timeout -r 1 [ 1261.874422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_timeout -r 1 [ 1261.904556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_timeout -r 1 [ 1261.938571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_timeout -r 1 [ 1261.973600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_timeout -r 1 [ 1262.008898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_timeout -r 1 [ 1262.033634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1262.162819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.185854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.212921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.252511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.279088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1262.317445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1262.336159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1262.357720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.386422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.422636] LNet: 61333:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.429433] LNet: 61333:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1262.459682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.522035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.603374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.668879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.733219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.788739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.827864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.479820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_error -r 1 [ 1263.513515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_error -r 1 [ 1263.551808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_error -r 1 [ 1263.604399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_error -r 1 [ 1263.660403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_error -r 1 [ 1263.735488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_error -r 1 [ 1263.812749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_error -r 1 [ 1263.880846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_error -r 1 [ 1263.955422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1264.222084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.252383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1264.282554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1264.309276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1264.338391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1264.375759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1264.402191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.453238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.482484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.518034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.554972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.577534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.603545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.643718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.732519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.798228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.963308] LNet: 61532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.969272] LNetError: 61532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.979715] LNet: Removed LNI 192.168.202.14@tcp [ 1272.041544] LNet: Removed LNI 192.168.202.14@tcp1 [ 1272.682669] Key type .llcrypt unregistered [ 1272.687767] Key type ._llcrypt unregistered [ 1294.985170] Key type ._llcrypt registered [ 1294.986562] Key type .llcrypt registered [ 1296.912163] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:17:37 (1789049857) [ 1298.528694] Key type .llcrypt unregistered [ 1298.530285] Key type ._llcrypt unregistered [ 1307.409889] Key type ._llcrypt registered [ 1307.414734] Key type .llcrypt registered [ 1307.730248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1307.789089] alg: No test for adler32 (adler32-zlib) [ 1308.753432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1308.789931] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1317.368805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1318.109283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_dropped -r 1 [ 1318.145556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_dropped -r 1 [ 1318.184431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1318.213785] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1318.435641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.482748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1318.528781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.573795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.607346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1318.649350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.685272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.721705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.758552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.784354] LNet: 62779:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.815622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.846862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.887807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.924869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.962654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1319.007780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1319.054769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.017572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_error -r 1 [ 1320.086855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_error -r 1 [ 1320.146561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1320.191393] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1320.566560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1320.606268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1320.668525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1320.731765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1320.795670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1320.870173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1320.923672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1320.979996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.036366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.051574] LNet: 62891:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.083480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.129551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.187598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.225574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.253910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.289739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1321.319786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1321.935911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_timeout -r 1 [ 1321.968720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_timeout -r 1 [ 1322.000516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1322.023739] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1322.265475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.341602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1322.375707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1322.450588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1322.484887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1322.535351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1322.611096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1322.646493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.675443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.726347] LNet: 63002:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.758610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.787451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.816222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.849506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.904849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.978951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1323.025567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1323.329663] LNet: 63043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.333920] LNetError: 63043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.349436] LNet: Removed LNI 192.168.202.14@tcp [ 1323.744399] Key type .llcrypt unregistered [ 1323.748136] Key type ._llcrypt unregistered [ 1341.636486] Key type ._llcrypt registered [ 1341.647217] Key type .llcrypt registered [ 1343.039360] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:18:24 (1789049904) [ 1344.000272] Key type .llcrypt unregistered [ 1344.002322] Key type ._llcrypt unregistered [ 1351.825157] Key type ._llcrypt registered [ 1351.828142] Key type .llcrypt registered [ 1352.036916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1352.050516] alg: No test for adler32 (adler32-zlib) [ 1353.142803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1353.230578] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1360.965162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1370.862970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.937113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1370.972623] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1372.256622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_dropped -r 1 [ 1372.308211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e remote_dropped -r 1 [ 1372.350476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_dropped -r 1 [ 1372.384310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e remote_dropped -r 1 [ 1372.428673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e remote_dropped -r 1 [ 1372.465311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e remote_dropped -r 1 [ 1372.519439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e remote_dropped -r 1 [ 1372.585510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e remote_dropped -r 1 [ 1372.663682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1372.692805] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1372.702540] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.114@tcp [ 1372.970444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1373.002601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1373.040906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1373.066776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1373.094372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1373.122361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.151916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1373.180761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1373.218214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1373.244176] LNet: 64508:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1373.256735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1373.279910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1373.312439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.342490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.367932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.392857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1373.418330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1373.446519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1373.984229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_error -r 1 [ 1374.036107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e remote_error -r 1 [ 1374.072952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_error -r 1 [ 1374.126794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e remote_error -r 1 [ 1374.180057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e remote_error -r 1 [ 1374.246164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e remote_error -r 1 [ 1374.307360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e remote_error -r 1 [ 1374.378209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e remote_error -r 1 [ 1374.425336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1374.448352] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1374.453873] LNet: Skipped 2 previous similar messages [ 1374.685624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1374.721808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1374.746758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1374.778434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1374.802835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1374.834516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1374.857434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1374.888289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1374.915855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1374.935554] LNet: 64633:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1374.950757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1374.987503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1375.036384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.068326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1375.118540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.189862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.222692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1375.251863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.791021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_timeout -r 1 [ 1375.818825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e remote_timeout -r 1 [ 1375.860194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_timeout -r 1 [ 1375.894608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e remote_timeout -r 1 [ 1375.933709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e remote_timeout -r 1 [ 1375.978155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e remote_timeout -r 1 [ 1376.008554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e remote_timeout -r 1 [ 1376.057831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e remote_timeout -r 1 [ 1376.119694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.114@tcp [ 1376.153192] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1376.420665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.463306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.491273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.509790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.529938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1376.555970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.579956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.618855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.646184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.664217] LNet: 64759:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.685142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.714150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.741721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.774786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.811595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.836766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1376.864827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1376.890446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1382.143617] LNet: 64836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.153796] LNetError: 64836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.165369] LNet: Removed LNI 192.168.202.14@tcp [ 1382.181699] LNet: Removed LNI 192.168.202.14@tcp1 [ 1382.701565] Key type .llcrypt unregistered [ 1382.703172] Key type ._llcrypt unregistered [ 1401.901704] Key type ._llcrypt registered [ 1401.911192] Key type .llcrypt registered [ 1403.643103] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:19:24 (1789049964) [ 1405.012772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1407.043508] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:19:28 (1789049968) [ 1409.060464] Key type .llcrypt unregistered [ 1409.064876] Key type ._llcrypt unregistered [ 1417.129437] Key type ._llcrypt registered [ 1417.137090] Key type .llcrypt registered [ 1417.509756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.537029] alg: No test for adler32 (adler32-zlib) [ 1418.546419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1418.594036] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1429.974489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1430.802600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e network_timeout -r 1 [ 1430.879488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e network_timeout -r 1 [ 1430.935972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1430.954977] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1431.345430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.380410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1431.416842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.462893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1431.519936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1431.581599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1431.636593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1431.701408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1431.741370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1431.761257] LNet: 66379:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.780618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1431.809712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1431.850392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1431.894558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1431.903591] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1431.906539] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1431.950539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.001708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.418470] LNet: 66418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.426378] LNetError: 66418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.455558] LNet: Removed LNI 192.168.202.14@tcp [ 1432.981734] Key type .llcrypt unregistered [ 1432.989189] Key type ._llcrypt unregistered [ 1458.841277] Key type ._llcrypt registered [ 1458.843675] Key type .llcrypt registered [ 1459.105861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.127338] alg: No test for adler32 (adler32-zlib) [ 1460.269698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.318839] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1470.834609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1483.140302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.165783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1483.193666] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1485.278345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e network_timeout -r 1 [ 1485.341397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e network_timeout -r 1 [ 1485.387638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e network_timeout -r 1 [ 1485.438307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e network_timeout -r 1 [ 1485.483243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e network_timeout -r 1 [ 1485.563295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e network_timeout -r 1 [ 1485.625387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e network_timeout -r 1 [ 1485.677933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e network_timeout -r 1 [ 1485.747578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1485.801718] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1485.989441] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1486.001400] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.114@tcp [ 1486.215419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1486.267186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1486.310799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1486.358893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1486.396940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1486.441777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1486.493546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.548893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.638308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.702383] LNet: 67306:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.761753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.852795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.936337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1487.008460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1487.056827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1487.074155] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1487.084880] LNet: Skipped 1 previous similar message [ 1487.096370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1489.120527] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1491.170329] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1492.960679] LNet: 67380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.968062] LNetError: 67380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.991157] LNet: Removed LNI 192.168.202.14@tcp [ 1493.997974] LNetError: 67380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.042706] LNet: Removed LNI 192.168.202.14@tcp1 [ 1494.694948] Key type .llcrypt unregistered [ 1494.698759] Key type ._llcrypt unregistered [ 1514.877801] Key type ._llcrypt registered [ 1514.887814] Key type .llcrypt registered [ 1516.721688] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:21:17 (1789050077) [ 1517.159654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1517.173894] alg: No test for adler32 (adler32-zlib) [ 1518.169275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.253074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.285533] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1519.227657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1519.270016] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1520.194870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 1520.283885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1520.374868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1520.445536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1520.501904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1520.557708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1520.594984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.14@tcp [ 1520.617290] LNetError: 68477:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.14@tcp to recovery [ 1521.311343] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1521.314356] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1523.360814] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1524.303642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.455445] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1527.929699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1527.981038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1528.092983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.121237] LNet: 68584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.132335] LNetError: 68584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.137898] LNet: Removed LNI 192.168.202.14@tcp [ 1528.155874] LNet: Removed LNI 192.168.202.14@tcp1 [ 1528.172275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.241793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.270677] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1529.225844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1529.251362] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1530.367772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 1530.446511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1530.515868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1530.615703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1530.721830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1530.803532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1530.851807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1530.905542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.14@tcp [ 1530.949506] LNetError: 68856:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.14@tcp to recovery [ 1531.299172] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1534.602590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1537.439422] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1537.443432] LNet: Skipped 1 previous similar message [ 1542.717969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1542.759716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1542.800751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.438667] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:21:50 (1789050110) [ 1549.748160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.774715] LNet: 69573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.784558] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.795931] LNetError: 69573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1549.819259] LNet: Removed LNI 192.168.202.14@tcp [ 1549.847500] LNet: Removed LNI 192.168.202.14@tcp1 [ 1549.865484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.914934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.944172] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1550.779580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.816913] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1551.868404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 1552.017643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1552.073628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1552.122322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1552.166199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1552.209403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1552.246520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.14@tcp --health 0 [ 1552.428097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1552.991306] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.14@tcp [ 1554.016058] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1554.040216] LNet: Skipped 1 previous similar message [ 1569.447578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.607633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1569.646194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1569.687857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.14@tcp --health 500 [ 1569.809633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1576.633639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1576.761758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.798081] LNet: 70073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.808401] LNetError: 70073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.814938] LNetError: 70073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1576.823442] LNet: Removed LNI 192.168.202.14@tcp [ 1576.837405] LNet: Removed LNI 192.168.202.14@tcp1 [ 1576.858964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.893439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.917674] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1577.908424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.940508] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1579.199106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 1579.241596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.320362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1579.361936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1579.411506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1579.439822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1579.486564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1579.524836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.14@tcp --health 0 [ 1580.961130] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1580.974337] LNet: Skipped 3 previous similar messages [ 1582.147249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1591.641266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1591.681591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1591.705829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1598.207246] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:22:39 (1789050159) [ 1598.772578] LNet: 71119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.785524] LNetError: 71119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.789360] LNetError: 71119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1598.797153] LNet: Removed LNI 192.168.202.14@tcp [ 1598.815370] LNet: Removed LNI 192.168.202.14@tcp1 [ 1599.700120] Key type .llcrypt unregistered [ 1599.702286] Key type ._llcrypt unregistered [ 1607.003510] Key type ._llcrypt registered [ 1607.005067] Key type .llcrypt registered [ 1607.237345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1607.246610] alg: No test for adler32 (adler32-zlib) [ 1608.184600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1608.241725] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1617.906090] Lustre: DEBUG MARKER: Initial discovery [ 1617.959584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.114@tcp [ 1620.291772] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1620.383678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1620.443439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.114@tcp [ 1620.488101] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1620.509051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1620.567634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.114@tcp -r 1 -m GET -l 3 [ 1620.589063] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1620.617957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.114@tcp -r 1 -m GET -e local_error [ 1620.644352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.114@tcp -r 1 -m PUT -l 6 [ 1622.167995] Lustre: DEBUG MARKER: Force oleg214-client.virtnet to discover 192.168.202.114@tcp (in background) [ 1622.337400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.114@tcp [ 1623.550116] Lustre: DEBUG MARKER: Force oleg214-server to discover 192.168.202.14@tcp [ 1625.439431] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1626.394605] Lustre: DEBUG MARKER: Wait for 71877 [ 1627.891573] Lustre: DEBUG MARKER: Finished wait on 71877 [ 1628.069477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1628.133158] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1628.146139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1628.437721] LNet: 72328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.445657] LNetError: 72328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.485251] LNet: Removed LNI 192.168.202.14@tcp [ 1630.002725] Key type .llcrypt unregistered [ 1630.005987] Key type ._llcrypt unregistered [ 1646.989875] Key type ._llcrypt registered [ 1646.992761] Key type .llcrypt registered [ 1648.198790] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:23:29 (1789050209) [ 1648.850314] Key type .llcrypt unregistered [ 1648.852793] Key type ._llcrypt unregistered [ 1655.724252] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1655.957176] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1655.967412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1655.972502] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1658.679668] Key type ._llcrypt registered [ 1658.684338] Key type .llcrypt registered [ 1658.922913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1658.941639] alg: No test for adler32 (adler32-zlib) [ 1659.954134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.037338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.062637] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1661.140996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1661.167496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1671.869536] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:23:52 (1789050232) [ 1672.058220] LNet: 74400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.067770] LNetError: 74400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1672.076346] LNet: Removed LNI 10.1.2.3@tcp [ 1672.098429] LNet: Removed LNI 192.168.202.14@tcp [ 1672.763824] Key type .llcrypt unregistered [ 1672.765995] Key type ._llcrypt unregistered [ 1680.845505] Key type ._llcrypt registered [ 1680.849177] Key type .llcrypt registered [ 1681.145711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1681.170530] alg: No test for adler32 (adler32-zlib) [ 1682.211455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.340957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.388369] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1683.470538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1683.489744] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1684.806634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.14@tcp [ 1685.856926] LNetError: Unexpected error -22 connecting to 192.168.202.14@tcp at host 192.168.202.14:988 [ 1692.408515] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:24:13 (1789050253) [ 1692.548890] LNet: 75524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.566819] LNetError: 75524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1692.574074] LNet: Removed LNI 10.1.2.3@tcp [ 1692.613594] LNet: Removed LNI 192.168.202.14@tcp [ 1693.328407] Key type .llcrypt unregistered [ 1693.330732] Key type ._llcrypt unregistered [ 1701.487672] Key type ._llcrypt registered [ 1701.491118] Key type .llcrypt registered [ 1701.735594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1701.749366] alg: No test for adler32 (adler32-zlib) [ 1702.787307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.920744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1702.943897] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1703.802373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1703.818789] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 1704.698683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.14@tcp1 --nid 192.168.202.14@tcp2 [ 1708.719516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp1 192.168.202.14@tcp2 [ 1708.734642] LNetError: 76365:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.14@tcp1 to 192.168.202.14@tcp2 - Route table may be misconfigured [ 1708.764887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp2 192.168.202.14@tcp1 [ 1715.998407] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:24:36 (1789050276) [ 1716.137312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.181545] LNet: 76932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.189167] LNetError: 76932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.199051] LNet: Removed LNI 192.168.202.14@tcp1 [ 1717.228561] LNet: Removed LNI 192.168.202.14@tcp2 [ 1717.307870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.358728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.379797] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1718.339957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1718.390767] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1719.452697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 1719.541529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.14@tcp [ 1719.617689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp1 192.168.202.14@tcp1 [ 1719.683705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.14@tcp -d 192.168.202.14@tcp -e network_timeout [ 1719.735772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -e network_timeout [ 1719.807711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -e network_timeout [ 1719.897601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -e network_timeout [ 1719.964992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1720.006508] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1720.036913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1720.170521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1720.278793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1720.419634] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1727.651521] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:24:48 (1789050288) [ 1727.943698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.970763] LNet: 77801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.983908] LNetError: 77801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.991436] LNetError: 77801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1728.000244] LNet: Removed LNI 192.168.202.14@tcp [ 1729.070482] LNet: Removed LNI 192.168.202.14@tcp1 [ 1729.103650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.254459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1729.547561] LNet: 77838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.070594] Key type .llcrypt unregistered [ 1730.071909] Key type ._llcrypt unregistered [ 1743.059901] Key type ._llcrypt registered [ 1743.062348] Key type .llcrypt registered [ 1744.213647] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:25:05 (1789050305) [ 1744.649463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1744.669233] alg: No test for adler32 (adler32-zlib) [ 1745.712593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.431879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1749.409757] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:25:10 (1789050310) [ 1749.576356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.604690] LNet: 78934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.641743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.739523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.768791] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1750.540368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1750.561261] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1751.676147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1751.736845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp1 [ 1751.804995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp1 [ 1759.564828] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:25:20 (1789050320) [ 1762.103516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg214-server [ 1764.234561] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:25:25 (1789050325) [ 1766.331277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg214-server [ 1768.299387] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:25:29 (1789050329) [ 1769.973324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg214-server [ 1771.936123] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:25:33 (1789050333) [ 1773.584780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg214-server [ 1775.518115] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:25:36 (1789050336) [ 1777.901108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg214-server [ 1779.657389] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:25:40 (1789050340) [ 1781.756945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg214-server [ 1783.911575] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:25:44 (1789050344) [ 1785.912635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg214-server [ 1787.516608] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:25:48 (1789050348) [ 1789.684811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg214-server [ 1791.254907] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:25:52 (1789050352) [ 1793.302250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg214-server [ 1795.734129] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:25:56 (1789050356) [ 1796.044057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.076157] LNet: 82812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.102118] LNetError: 82812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.111593] LNet: Removed LNI 192.168.202.14@tcp [ 1797.160840] LNet: Removed LNI 192.168.202.14@tcp1 [ 1797.237358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.375425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1797.440952] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1798.944891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1799.104678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1799.354738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.393335] LNet: 82960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.411790] LNetError: 82960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.428300] LNetError: 82960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1799.468623] LNet: Removed LNI 192.168.202.14@tcp [ 1799.506374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.600789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1799.623271] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1800.683579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1801.012218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.043354] LNetError: 83105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.103168] LNet: Removed LNI 192.168.202.14@tcp [ 1802.206544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.296232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.323912] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1803.574199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1803.668517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1803.872755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.892591] LNet: 83253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.896736] LNet: 83253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1803.900583] LNetError: 83253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.910839] LNet: Removed LNI 192.168.202.14@tcp [ 1804.013122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.146557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1804.192033] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1805.013492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1805.117325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.197077] LNet: Removed LNI 192.168.202.14@tcp [ 1806.291371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.412627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.464427] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1807.436909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1807.591942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1807.830261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.880079] LNet: Removed LNI 192.168.202.14@tcp [ 1808.919367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.964325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1808.991383] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1810.211802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1810.636258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.696098] LNet: 83691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.716681] LNet: 83691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1810.747561] LNetError: 83691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.759554] LNetError: 83691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1810.785839] LNet: Removed LNI 192.168.202.14@tcp [ 1810.861767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.939786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.960693] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1812.173362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1812.259758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1812.546880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.610372] LNet: Removed LNI 192.168.202.14@tcp [ 1813.654253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.762701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1813.792804] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1814.932739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1815.183621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.293944] LNet: Removed LNI 192.168.202.14@tcp [ 1816.374649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.493916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1816.531315] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1817.665357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1817.862783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1818.079807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.177040] LNet: Removed LNI 192.168.202.14@tcp [ 1819.200685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.300994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1819.346262] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1820.409935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1820.678175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.696162] LNet: 84277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.700758] LNet: 84277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1820.706884] LNetError: 84277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.713222] LNetError: 84277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1821.736852] LNet: Removed LNI 192.168.202.14@tcp [ 1821.831633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.896932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.921603] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1822.348935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1822.385827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1822.584344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.670573] LNet: Removed LNI 192.168.202.14@tcp [ 1823.737360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.864795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1823.894216] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1824.871541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 1825.107429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.177297] LNet: Removed LNI 192.168.202.14@tcp [ 1826.254097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.350804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1826.377791] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1827.465676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1827.539965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.567613] LNet: Removed LNI 192.168.202.14@tcp [ 1827.597307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.697701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.726024] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1828.462194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1828.586671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.665259] LNet: Removed LNI 192.168.202.14@tcp [ 1828.721694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.782963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1828.903165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.953454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.049963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1829.134772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.174784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.239679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1829.326956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.361445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.424171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1829.469568] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1830.406092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.428440] LNet: Removed LNI 192.168.202.14@tcp [ 1830.445972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.525168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1838.405952] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:26:39 (1789050399) [ 1838.688419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.720360] LNet: 85592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.726279] LNet: 85592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1838.751462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.807645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1838.844664] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/180] [ 1840.420621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.447721] LNetError: 85730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.457293] LNetError: 85730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1840.486924] LNet: Removed LNI 192.168.202.14@tcp231 [ 1840.509868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.595089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1840.622607] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/180] [ 1840.628077] LNet: 85738:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1841.337743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1841.412860] LNet: Removed LNI 192.168.202.14@tcp231 [ 1841.455514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1841.480515] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/180] [ 1843.350926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1843.370608] LNet: Removed LNI 192.168.202.14@tcp231 [ 1843.381591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.441416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1843.464951] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/0] [ 1843.473079] LNet: 85890:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1844.139453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1844.231659] LNet: Removed LNI 192.168.202.14@tcp231 [ 1844.291571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1844.334651] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/0] [ 1845.687913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.711769] LNet: Removed LNI 192.168.202.14@tcp231 [ 1845.729900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.783838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1845.803833] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/60] [ 1845.809610] LNet: 86042:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1846.373223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1846.406070] LNet: Removed LNI 192.168.202.14@tcp231 [ 1846.439212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1846.482924] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/60] [ 1847.430757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.478864] LNet: Removed LNI 192.168.202.14@tcp231 [ 1847.507279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.565311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1847.583985] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/180] [ 1848.326587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1848.369755] LNet: Removed LNI 192.168.202.14@tcp231 [ 1848.401971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1848.437358] LNet: Added LNI 192.168.202.14@tcp231 [8/256/0/180] [ 1857.216131] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:26:58 (1789050418) [ 1857.475913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.527552] LNet: Removed LNI 192.168.202.14@tcp231 [ 1857.567623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.697846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1857.750431] LNet: Added LNI 192.168.202.14@tcp232 [8/256/0/180] [ 1858.451802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1858.483569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1858.516416] LNet: Removed LNI 192.168.202.14@tcp232 [ 1858.534637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1858.560069] LNet: Added LNI 192.168.202.14@tcp232 [8/256/0/180] [ 1859.555518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1859.604666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1859.645780] LNet: Removed LNI 192.168.202.14@tcp232 [ 1859.673629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1859.725272] LNet: Added LNI 192.168.202.14@tcp232 [8/256/0/180] [ 1860.775741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1860.828257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1860.868480] LNet: Removed LNI 192.168.202.14@tcp232 [ 1860.889687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1860.917255] LNet: Added LNI 192.168.202.14@tcp232 [8/256/0/180] [ 1861.708740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1861.745697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1861.764535] LNet: Removed LNI 192.168.202.14@tcp232 [ 1861.772860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1867.814678] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:27:09 (1789050429) [ 1868.793538] Key type .llcrypt unregistered [ 1868.795421] Key type ._llcrypt unregistered [ 1875.870564] Key type ._llcrypt registered [ 1875.875995] Key type .llcrypt registered [ 1876.277976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1876.305595] alg: No test for adler32 (adler32-zlib) [ 1877.384354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1877.455623] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1886.248638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 1898.424370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.491534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1898.507352] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1899.985287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_interrupt -r 2 [ 1900.022805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_interrupt -r 2 [ 1900.063885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_interrupt -r 2 [ 1900.132371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_interrupt -r 2 [ 1900.232807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_interrupt -r 2 [ 1900.322259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_interrupt -r 2 [ 1900.392335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_interrupt -r 2 [ 1900.459441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_interrupt -r 2 [ 1900.540389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1900.575844] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1900.589832] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1900.816659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1900.848326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1900.866548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1900.889341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1900.921970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1900.951855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1900.982989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1901.015289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1901.045877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1901.081927] LNet: 88696:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1901.125104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1901.184745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1901.226183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1901.260821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1901.302205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1901.347587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1901.999870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_dropped -r 2 [ 1902.039035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_dropped -r 2 [ 1902.094356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_dropped -r 2 [ 1902.149316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_dropped -r 2 [ 1902.204600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_dropped -r 2 [ 1902.255446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_dropped -r 2 [ 1902.306339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_dropped -r 2 [ 1902.365729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_dropped -r 2 [ 1902.406732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1902.434520] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1902.774820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1902.812324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1902.846281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1902.879232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1902.906471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1902.934868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1902.951690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1902.968364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1902.981238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1902.989334] LNet: 88818:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1902.998529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1903.018132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1903.033728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1903.053290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1903.079071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1903.103510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1903.736718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_aborted -r 2 [ 1903.773657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_aborted -r 2 [ 1903.801769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_aborted -r 2 [ 1903.831172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_aborted -r 2 [ 1903.866696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_aborted -r 2 [ 1903.901833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_aborted -r 2 [ 1903.932345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_aborted -r 2 [ 1903.991970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_aborted -r 2 [ 1904.066691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1904.093948] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1904.314879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1904.344284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1904.391914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1904.443886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1904.483337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1904.518160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1904.558477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1904.580560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1904.615624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1904.643949] LNet: 88940:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1904.668990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1904.704679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1904.730291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1904.760332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1904.790830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1904.835245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1905.603039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_no_route -r 2 [ 1905.640578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_no_route -r 2 [ 1905.670310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_no_route -r 2 [ 1905.703953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_no_route -r 2 [ 1905.730758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_no_route -r 2 [ 1905.759755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_no_route -r 2 [ 1905.792985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_no_route -r 2 [ 1905.821801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_no_route -r 2 [ 1905.852683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1906.123851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1906.189935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1906.226658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1906.255925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1906.282618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1906.310343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1906.342816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1906.378785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1906.416169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1906.443364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1906.474814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1906.499646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1906.531967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1906.564164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1906.606249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1907.252509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e local_timeout -r 2 [ 1907.288498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e local_timeout -r 2 [ 1907.321783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e local_timeout -r 2 [ 1907.352258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e local_timeout -r 2 [ 1907.372094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e local_timeout -r 2 [ 1907.409542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e local_timeout -r 2 [ 1907.435964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e local_timeout -r 2 [ 1907.497823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e local_timeout -r 2 [ 1907.530948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1907.554051] LNet: There was an unexpected network error while writing to 192.168.202.114: rc = -22 [ 1907.561756] LNet: Skipped 1 previous similar message [ 1907.730082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1907.748471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1907.768294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1907.789308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1907.810422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1907.834582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1907.876780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1907.914139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.956700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1907.973758] LNet: 89185:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1907.988927] LNet: 89185:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1908.005936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1908.030857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1908.074891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1908.103932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1908.145983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1908.177829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1908.736942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp -m GET -e remote_dropped -r 2 [ 1908.768899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp -d 192.168.202.114@tcp1 -m GET -e remote_dropped -r 2 [ 1908.801295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp -m GET -e remote_dropped -r 2 [ 1908.838094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp -d 192.168.202.14@tcp1 -m GET -e remote_dropped -r 2 [ 1908.865461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp -m GET -e remote_dropped -r 2 [ 1908.893988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.14@tcp1 -d 192.168.202.114@tcp1 -m GET -e remote_dropped -r 2 [ 1908.939167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp -m GET -e remote_dropped -r 2 [ 1908.996147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.14@tcp1 -d 192.168.202.14@tcp1 -m GET -e remote_dropped -r 2 [ 1909.051228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.14@tcp 192.168.202.114@tcp [ 1909.076280] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.114@tcp [ 1909.286626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1909.309588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1909.341270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1909.368352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1909.388952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1909.424804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1909.452696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1909.480914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1909.518148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1909.536442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1909.561583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1909.593386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1909.618779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1909.672532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1909.729570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1916.002168] LNet: 89382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.010134] LNetError: 89382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.023335] LNet: Removed LNI 192.168.202.14@tcp [ 1917.030685] LNetError: 89382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.058456] LNet: Removed LNI 192.168.202.14@tcp1 [ 1917.481593] Key type .llcrypt unregistered [ 1917.483940] Key type ._llcrypt unregistered [ 1933.538681] Key type ._llcrypt registered [ 1933.544869] Key type .llcrypt registered [ 1934.786404] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:28:16 (1789050496) [ 1935.071708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1935.083648] alg: No test for adler32 (adler32-zlib) [ 1936.036324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.099978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1936.132692] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1937.105447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1937.157207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.14@tcp [ 1937.191639] LNetError: 90360:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.14@tcp to recovery [ 1937.306742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.14@tcp [ 1937.448070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1944.121212] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:28:25 (1789050505) [ 1944.241684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.281799] LNet: 90943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.294681] LNetError: 90943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.334308] LNet: Removed LNI 192.168.202.14@tcp [ 1944.381900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.476574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.514802] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1945.320492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1945.352407] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 1946.572657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.14@tcp [ 1946.596344] LNetError: 91216:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.202.14@tcp to recovery [ 1947.487370] LNet: 1 local NIs in recovery (showing 1): 192.168.202.14@tcp [ 1947.497412] LNet: There was an unexpected network error while writing to 192.168.202.14: rc = -22 [ 1947.893434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.14@tcp [ 1957.069974] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:28:38 (1789050518) [ 1958.766609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg214-server [ 1960.584728] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:28:41 (1789050521) [ 1962.825901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg214-server [ 1964.711675] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:28:45 (1789050525) [ 1967.344406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg214-server [ 1969.525580] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:28:50 (1789050530) [ 1969.777796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.842026] LNet: 92848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.864412] LNetError: 92848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.883892] LNet: Removed LNI 192.168.202.14@tcp [ 1969.892853] LNet: Removed LNI 192.168.202.14@tcp1 [ 1969.916735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.986238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.026265] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1970.832155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1970.937099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1970.989323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1971.110700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1971.151685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1971.225843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1977.716627] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:28:58 (1789050538) [ 1977.850642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.889273] LNet: 93575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.899664] LNetError: 93575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1977.913561] LNetError: 93575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1977.963568] LNet: Removed LNI 192.168.202.14@tcp [ 1978.035033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.100383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1978.127492] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1984.975758] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:29:06 (1789050546) [ 1985.225322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.260040] LNet: 94284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.264619] LNetError: 94284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.279659] LNet: Removed LNI 192.168.202.14@tcp [ 1985.304217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.376199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.404362] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 1992.583909] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:29:13 (1789050553) [ 1994.023381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1995.634149] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:29:16 (1789050556) [ 1996.202777] LNet: 95319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.207766] LNetError: 95319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.229287] LNet: Removed LNI 192.168.202.14@tcp [ 1996.953072] Key type .llcrypt unregistered [ 1996.957362] Key type ._llcrypt unregistered [ 2004.327321] Key type ._llcrypt registered [ 2004.333122] Key type .llcrypt registered [ 2004.616237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2004.626520] alg: No test for adler32 (adler32-zlib) [ 2005.545977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2005.574418] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2013.288883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 2018.845236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.114@tcp [ 2019.513471] LNet: 95734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.524924] LNetError: 95734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.547251] LNet: Removed LNI 192.168.202.14@tcp [ 2020.028620] Key type .llcrypt unregistered [ 2020.032450] Key type ._llcrypt unregistered [ 2031.132854] Key type ._llcrypt registered [ 2031.137889] Key type .llcrypt registered [ 2032.312744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2033.767220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2035.277520] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:29:56 (1789050596) [ 2036.588756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2038.168747] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:29:59 (1789050599) [ 2039.973963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg214-server [ 2041.390860] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:30:02 (1789050602) [ 2042.924641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg214-server [ 2044.662298] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:30:05 (1789050605) [ 2045.834248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.057041] alg: No test for adler32 (adler32-zlib) [ 2047.088992] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2047.314047] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2048.984400] Key type lgssc registered [ 2050.033621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.439532] Key type lgssc unregistered [ 2068.671591] LNet: 98904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.680282] LNetError: 98904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.694911] LNet: Removed LNI 192.168.202.14@tcp [ 2069.253484] Key type .llcrypt unregistered [ 2069.255336] Key type ._llcrypt unregistered [ 2081.879523] Key type ._llcrypt registered [ 2081.881572] Key type .llcrypt registered [ 2083.195221] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:30:44 (1789050644) [ 2083.465542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2083.485152] alg: No test for adler32 (adler32-zlib) [ 2084.318475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.362123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2084.405274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2084.477518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2084.512079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2091.221938] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:30:52 (1789050652) [ 2091.599156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.647358] LNet: 100319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.692941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.759178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2091.784718] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2092.756835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.795882] LNet: 100463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.814178] LNetError: 100463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.830862] LNet: Removed LNI 192.168.202.14@tcp [ 2092.864191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.950597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2092.991186] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2093.523469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.541280] LNetError: 100491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2093.550125] LNet: Removed LNI 192.168.202.14@tcp [ 2093.566283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.619849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2093.647261] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2100.436517] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:31:01 (1789050661) [ 2100.503424] LNet: 101074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.507242] LNet: 101074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2100.518231] LNetError: 101074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.529960] LNet: Removed LNI 192.168.202.14@tcp [ 2101.076514] Key type .llcrypt unregistered [ 2101.079516] Key type ._llcrypt unregistered [ 2107.357199] Key type ._llcrypt registered [ 2107.359322] Key type .llcrypt registered [ 2107.538566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2107.563189] alg: No test for adler32 (adler32-zlib) [ 2108.590962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.692549] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2108.693959] LNet: *** cfs_fail_loc=e003, val=100*** [ 2109.380914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2109.401975] LNet: *** cfs_fail_loc=e003, val=100*** [ 2109.404702] LNet: Skipped 2 previous similar messages [ 2109.419298] Ping reply received with mapping: [ 2109.422954] found metadata reply NI[192.168.202.14@tcp] [ 2109.426480] 0e ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2109.431088] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2109.435416] 5a 5a 5a 5a 5a 5a 5a 5a [ 2109.608688] LNet: 101476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.616082] LNetError: 101476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.630654] LNet: Removed LNI 192.168.202.14@tcp [ 2110.315458] Key type .llcrypt unregistered [ 2110.322324] Key type ._llcrypt unregistered [ 2122.857747] Key type ._llcrypt registered [ 2122.861444] Key type .llcrypt registered [ 2124.205945] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:31:25 (1789050685) [ 2124.970328] Key type .llcrypt unregistered [ 2124.973340] Key type ._llcrypt unregistered [ 2131.366241] Key type ._llcrypt registered [ 2131.374180] Key type .llcrypt registered [ 2131.595945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.614352] alg: No test for adler32 (adler32-zlib) [ 2132.744880] LNetError: 102514:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2133.303742] Key type .llcrypt unregistered [ 2133.308594] Key type ._llcrypt unregistered [ 2138.241690] Key type ._llcrypt registered [ 2138.244926] Key type .llcrypt registered [ 2139.598850] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:31:40 (1789050700) [ 2140.072380] Key type .llcrypt unregistered [ 2140.076543] Key type ._llcrypt unregistered [ 2146.280347] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2146.287143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2148.749577] Key type ._llcrypt registered [ 2148.751038] Key type .llcrypt registered [ 2148.931593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2148.941813] alg: No test for adler32 (adler32-zlib) [ 2149.823326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.871922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2157.960637] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:31:59 (1789050719) [ 2158.093702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.112805] LNet: 104282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.123277] LNetError: 104282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2158.147908] LNet: Removed LNI 10.1.2.3@tcp [ 2158.172327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.897971] Key type .llcrypt unregistered [ 2158.903737] Key type ._llcrypt unregistered [ 2164.764093] Key type ._llcrypt registered [ 2164.766168] Key type .llcrypt registered [ 2164.940690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2164.955614] alg: No test for adler32 (adler32-zlib) [ 2165.880715] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2165.947323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2166.054287] LNet: 104571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.058797] LNetError: 104571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2167.083129] LNet: Removed LNI 192.168.202.14@tcp [ 2167.638495] Key type .llcrypt unregistered [ 2167.641281] Key type ._llcrypt unregistered [ 2177.435717] Key type ._llcrypt registered [ 2177.437799] Key type .llcrypt registered [ 2178.543384] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:32:19 (1789050739) [ 2179.161255] Key type .llcrypt unregistered [ 2179.165665] Key type ._llcrypt unregistered [ 2185.048697] Key type ._llcrypt registered [ 2185.053229] Key type .llcrypt registered [ 2185.245487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2185.259390] alg: No test for adler32 (adler32-zlib) [ 2192.186902] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:32:33 (1789050753) [ 2192.446055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2192.507257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2192.561346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.581774] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2193.506502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.548485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2193.587708] LNet: Removed LNI 192.168.202.14@tcp [ 2193.966834] LNet: 106309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.522331] Key type .llcrypt unregistered [ 2194.524679] Key type ._llcrypt unregistered [ 2207.242132] Key type ._llcrypt registered [ 2207.243548] Key type .llcrypt registered [ 2208.480185] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:32:49 (1789050769) [ 2209.066843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2209.088836] alg: No test for adler32 (adler32-zlib) [ 2210.112720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.201288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2210.247057] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2210.991517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2211.041726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.14@tcp [ 2211.083149] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2211.089596] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2211.094371] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2211, 0, 0 [ 2211.101157] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2211.105134] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2211.110506] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2211.115621] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2211, 0, 0 [ 2211.121215] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2211.122850] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2211.126933] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2211.130479] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2211, 0, 2261 [ 2211.134647] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2211.136981] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2211.140610] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2211.143995] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2211, 0, 2261 [ 2211.150052] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2217.588181] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:32:59 (1789050779) [ 2218.059790] LNet: 107846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.068744] LNetError: 107846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.088195] LNet: Removed LNI 192.168.202.14@tcp [ 2218.836812] Key type .llcrypt unregistered [ 2218.840760] Key type ._llcrypt unregistered [ 2225.649416] Key type ._llcrypt registered [ 2225.651067] Key type .llcrypt registered [ 2225.828128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2225.869677] alg: No test for adler32 (adler32-zlib) [ 2226.831169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2226.846929] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2234.276835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 2243.585882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.655532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2243.675089] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 2244.451772] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2244.581252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2244.584729] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2244.590823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2246.978243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2247.005369] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2258.546290] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:33:39 (1789050819) [ 2258.673130] LNet: 109344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.692336] LNetError: 109344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.705285] LNet: Removed LNI 192.168.202.14@tcp [ 2259.743604] LNet: Removed LNI 192.168.202.14@tcp1 [ 2259.748823] LNetError: 109344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2259.761967] LNetError: 109344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2259.775717] LNet: Removed LNI 10.1.2.3@tcp99 [ 2260.475275] Key type .llcrypt unregistered [ 2260.478970] Key type ._llcrypt unregistered [ 2266.847259] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2266.849641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2269.246436] Key type ._llcrypt registered [ 2269.248418] Key type .llcrypt registered [ 2269.423588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2269.464769] alg: No test for adler32 (adler32-zlib) [ 2270.424232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.490437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2270.502574] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2271.002627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2271.021536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.111735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2273.148516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.172332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2273.187328] LNetError: 110091:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2273.219748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.262839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2273.790778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2273.841462] LNet: 110111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.845809] LNetError: 110111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2273.850636] LNet: Removed LNI 10.1.2.3@tcp [ 2274.924020] LNet: Removed LNI 192.168.202.14@tcp [ 2275.902520] Key type .llcrypt unregistered [ 2275.904982] Key type ._llcrypt unregistered [ 2289.058737] Key type ._llcrypt registered [ 2289.060673] Key type .llcrypt registered [ 2290.902775] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:34:11 (1789050851) [ 2291.781771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2291.797631] alg: No test for adler32 (adler32-zlib) [ 2292.880398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.964605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.006300] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2294.299454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2294.450275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg214-client@tcp [ 2301.817552] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:34:22 (1789050862) [ 2302.060611] LNet: 111634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.101196] LNetError: 111634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.142698] LNet: Removed LNI 192.168.202.14@tcp [ 2302.883385] Key type .llcrypt unregistered [ 2302.884676] Key type ._llcrypt unregistered [ 2309.475429] Key type ._llcrypt registered [ 2309.477024] Key type .llcrypt registered [ 2309.708392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2309.735146] alg: No test for adler32 (adler32-zlib) [ 2310.756630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2310.796075] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2310.833948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2317.272461] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:34:38 (1789050878) [ 2318.861513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2320.471341] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:34:41 (1789050881) [ 2322.308174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2324.171293] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:34:45 (1789050885) [ 2324.253733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.267847] LNet: 113055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.277028] LNetError: 113055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.319207] LNet: Removed LNI 192.168.202.14@tcp [ 2324.343470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.457530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.478433] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2325.356607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2325.415740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2331.579439] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:34:53 (1789050893) [ 2331.793639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.808402] LNet: 113770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.814719] LNetError: 113770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.830805] LNet: Removed LNI 192.168.202.14@tcp [ 2331.850879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.926895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2331.953898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2332.522340] Key type .llcrypt unregistered [ 2332.526063] Key type ._llcrypt unregistered [ 2342.798937] Key type ._llcrypt registered [ 2342.803858] Key type .llcrypt registered [ 2344.063278] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:35:05 (1789050905) [ 2344.511279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.525984] alg: No test for adler32 (adler32-zlib) [ 2345.504243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.588884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.622533] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2346.642756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2346.691767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2351.902444] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:35:13 (1789050913) [ 2352.058192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.070556] LNet: 115324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.074618] LNetError: 115324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.084150] LNet: Removed LNI 192.168.202.14@tcp [ 2352.114615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.166755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2352.216437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2357.288452] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:35:18 (1789050918) [ 2357.383777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.415864] LNet: 115896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2357.452341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.531615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2357.940267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2357.967602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2358.005591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2358.265759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2358.292821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2358.349963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.388191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.435158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2358.901385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2358.986735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2359.121559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2359.505751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2359.536772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2366.721796] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:35:27 (1789050927) [ 2366.863674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.926331] LNet: 116530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.947496] LNet: 116530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2367.030408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.129639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.192394] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2368.414334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2368.455705] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2369.685628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2369.739681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2369.944925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.959119] LNet: 116816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.962861] LNetError: 116816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.966157] LNet: Removed LNI 192.168.202.14@tcp [ 2369.980864] LNet: Removed LNI 192.168.202.14@tcp2 [ 2369.994556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2370.036747] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2370.040940] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2370.044922] LNet: 116823:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.532293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2370.738322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.808316] LNet: Removed LNI 192.168.202.14@tcp [ 2371.816586] LNetError: 116860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2371.830281] LNetError: 116860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2371.858797] LNet: Removed LNI 192.168.202.14@tcp2 [ 2371.930438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.987021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2372.033072] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2372.836214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2372.864798] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2373.868276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2373.947719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2374.014822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2374.254452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.275085] LNet: 117144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.279332] LNet: 117144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2374.285916] LNetError: 117144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.327161] LNet: Removed LNI 192.168.202.14@tcp [ 2376.358249] LNet: Removed LNI 192.168.202.14@tcp2 [ 2376.384544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.510660] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2376.519758] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2376.530681] LNet: 117151:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.457769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2377.732268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2377.759408] LNet: Removed LNI 192.168.202.14@tcp [ 2378.799337] LNet: Removed LNI 192.168.202.14@tcp2 [ 2378.855722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.953384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.989584] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2379.861611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2379.882384] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2380.917751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2381.014529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2381.355491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.378506] LNetError: 117468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.382366] LNetError: 117468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2381.388509] LNet: Removed LNI 192.168.202.14@tcp [ 2381.422812] LNet: Removed LNI 192.168.202.14@tcp2 [ 2381.467860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.560839] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2381.571364] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2381.577925] LNet: 117474:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2382.328358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2382.555941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.581112] LNet: 117511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.585922] LNet: 117511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2383.647227] LNet: Removed LNI 192.168.202.14@tcp [ 2383.660921] LNet: Removed LNI 192.168.202.14@tcp2 [ 2383.701744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.753329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.791958] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2384.583234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2384.617469] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2385.610989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2385.676535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2385.742841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2386.023477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.103260] LNet: Removed LNI 192.168.202.14@tcp [ 2388.134069] LNet: Removed LNI 192.168.202.14@tcp2 [ 2388.158767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.219757] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2388.226268] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2388.230320] LNet: 117803:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.893840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2389.051860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.075223] LNet: Removed LNI 192.168.202.14@tcp [ 2390.133285] LNet: Removed LNI 192.168.202.14@tcp2 [ 2390.150942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.206744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.232081] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2391.222709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2391.259651] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2392.094704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2392.144566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2392.221532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2392.627877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.673376] LNetError: 118137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.684446] LNetError: 118137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2392.701439] LNet: Removed LNI 192.168.202.14@tcp [ 2392.714430] LNet: Removed LNI 192.168.202.14@tcp2 [ 2392.742790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.799365] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2392.804758] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2392.811597] LNet: 118143:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.434646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2393.744623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.783682] LNet: Removed LNI 192.168.202.14@tcp [ 2394.794872] LNet: Removed LNI 192.168.202.14@tcp2 [ 2394.827096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.881437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.911820] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2395.647347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2395.663400] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2396.465575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2396.533289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2396.572583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2396.661683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2397.014780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.111769] LNet: Removed LNI 192.168.202.14@tcp [ 2399.155643] LNet: Removed LNI 192.168.202.14@tcp2 [ 2399.193593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.290836] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2399.299831] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2399.824329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2406.964324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2408.677296] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:36:09 (1789050969) [ 2409.045975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.093796] LNet: 119262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2409.100550] LNet: 119262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2409.107560] LNetError: 119262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2409.112694] LNetError: 119262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2409.119634] LNet: Removed LNI 192.168.202.14@tcp [ 2410.153380] LNet: Removed LNI 192.168.202.14@tcp2 [ 2410.182467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2410.611915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.634109] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2411.415536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2411.436904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2412.157623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.175875] LNet: Removed LNI 10.1.2.3@tcp [ 2412.189518] LNet: Removed LNI 192.168.202.14@tcp [ 2412.207283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.268735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2412.291741] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2412.295490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2412.722926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2412.751734] LNet: Removed LNI 192.168.202.14@tcp [ 2412.768377] LNet: Removed LNI 10.1.2.3@tcp [ 2419.057944] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:36:20 (1789050980) [ 2419.202461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2419.257364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.340453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2419.386316] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2420.000742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2420.031518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2420.065917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.118798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.142837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.176559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.221593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.268665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.324095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.383484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.440297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.482521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2420.730936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2427.893926] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 10:36:28 (1789050988) [ 2428.006424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.100550] LNet: Removed LNI 192.168.202.14@tcp [ 2429.128870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.204416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.243248] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2430.036951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2430.061721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2430.096740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.133694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.163729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.194157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.224523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.249983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.280172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.309267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.339961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.363900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.440847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2430.513522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.541585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.584715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.618403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.664332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.707812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.738115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.770129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.798527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2430.832399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.14@tcp [ 2437.569813] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:36:38 (1789050998) [ 2437.849930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2438.952398] LNet: Removed LNI 192.168.202.14@tcp [ 2438.982812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2439.161644] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2439.347156] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2439.351860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2439.358382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2441.690357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2441.726554] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2442.534268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2442.551052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2444.633763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2444.654915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2444.720663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.14@tcp -m PUT -e local_timeout -r 1 [ 2444.792132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2444.860335] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2444.864183] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2445.181993] LNetError: 121702:0:(lib-move.c:4794:lnet_parse()) 192.168.202.14@tcp, src 192.168.202.14@tcp: Dropping PUT (error -108 looking up sender) [ 2445.919911] LNet: 122160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.936286] LNet: 122160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2445.952872] LNetError: 122160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2445.957202] LNetError: 122160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2445.969331] LNet: Removed LNI 10.1.2.3@tcp [ 2447.018649] LNet: Removed LNI 192.168.202.14@tcp [ 2447.459838] Key type .llcrypt unregistered [ 2447.464603] Key type ._llcrypt unregistered [ 2458.546621] Key type ._llcrypt registered [ 2458.556461] Key type .llcrypt registered [ 2459.748646] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:37:00 (1789051020) [ 2460.255663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2460.296572] alg: No test for adler32 (adler32-zlib) [ 2461.176961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.539316] LNet: 123010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.318359] Key type .llcrypt unregistered [ 2462.320182] Key type ._llcrypt unregistered [ 2469.670464] Key type ._llcrypt registered [ 2469.675538] Key type .llcrypt registered [ 2469.973958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2470.010918] alg: No test for adler32 (adler32-zlib) [ 2470.960609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2470.982931] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2478.618895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 2478.878896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2479.494950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.114@tcp [ 2479.708583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.734786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.758481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.799301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.841964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2479.879523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.906979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.949763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.981521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2479.997965] LNet: 123460:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2480.016543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.059963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.115193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.177948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.213190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.225769] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.114@tcp [ 2480.248744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2481.007696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2481.335878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2481.367678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2481.424942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2481.449350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2481.483341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2481.519459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2481.560925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2481.611056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2481.667251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2481.714501] LNet: 123579:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2481.742646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2481.798295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2481.846794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2481.917803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2481.962523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2482.026969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2482.633804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.114@tcp [ 2482.846774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2482.896914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2482.925645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2482.946871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2482.969921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2482.991351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.014515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.039482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.063244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2483.077373] LNet: 123698:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2483.097647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2483.141627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2483.185245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2483.221584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2483.256132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2483.304203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2483.936454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2484.185951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.222371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2484.259358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2484.290770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2484.328287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2484.359810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2484.392939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2484.422190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2484.447783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.465751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.505917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.541607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.571714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.607586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.644801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.942733] LNet: 123864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.952213] LNetError: 123864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.970715] LNet: Removed LNI 192.168.202.14@tcp [ 2485.420936] Key type .llcrypt unregistered [ 2485.422813] Key type ._llcrypt unregistered [ 2502.425882] Key type ._llcrypt registered [ 2502.429140] Key type .llcrypt registered [ 2503.665475] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:37:44 (1789051064) [ 2504.176643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2504.200923] alg: No test for adler32 (adler32-zlib) [ 2505.105889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2505.525965] LNet: 124750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.076476] Key type .llcrypt unregistered [ 2506.079084] Key type ._llcrypt unregistered [ 2511.708085] Key type ._llcrypt registered [ 2511.709832] Key type .llcrypt registered [ 2511.905393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2511.931992] alg: No test for adler32 (adler32-zlib) [ 2512.865448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2512.881927] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2520.486506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.114@tcp [ 2530.835514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.933616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2530.976904] LNet: Added LNI 192.168.202.14@tcp1 [8/256/0/180] [ 2532.101292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2532.688965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.114@tcp [ 2532.936170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2532.974765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2533.004742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2533.058937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2533.094785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2533.123818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2533.154258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2533.194589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2533.226309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2533.247939] LNet: 125402:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2533.284981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2533.337505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2533.392209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2533.425030] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.114@tcp [ 2533.463900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2533.529397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2533.590785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.159524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2534.407758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2534.438624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2534.473231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2534.508345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2534.536760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2534.561427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2534.591893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.636335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.709328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.743210] LNet: 125521:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.784361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.810563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.839384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.876841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.911990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.950614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2535.572420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.114@tcp [ 2535.899878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2535.929792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2535.981665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2536.027172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2536.048944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2536.086503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2536.124160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2536.153273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2536.189477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2536.216649] LNet: 125641:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2536.238533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2536.279487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2536.313976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2536.360568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2536.384503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2536.437287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2537.031737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2537.364698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2537.398000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2537.433539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2537.465182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2537.506422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2537.546488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2537.578084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2537.624614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2537.663557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2537.707265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2537.744913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2537.781274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2537.828554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2537.866450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2537.913554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.733775] LNet: 125841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.741985] LNetError: 125841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.775111] LNet: Removed LNI 192.168.202.14@tcp [ 2543.780298] LNetError: 125841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2543.806205] LNet: Removed LNI 192.168.202.14@tcp1 [ 2544.306615] Key type .llcrypt unregistered [ 2544.309264] Key type ._llcrypt unregistered [ 2562.650539] Key type ._llcrypt registered [ 2562.655583] Key type .llcrypt registered [ 2564.083101] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:38:45 (1789051125) [ 2564.594788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.615213] alg: No test for adler32 (adler32-zlib) [ 2565.681860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.754293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2565.800155] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2566.549250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2566.571890] LNet: Added LNI 192.168.202.14@tcp2 [8/256/0/180] [ 2567.422678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.14@tcp [ 2567.490652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2568.543128] LNet: Removed LNI 192.168.202.14@tcp [ 2574.515912] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:38:55 (1789051135) [ 2575.656440] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2577.114926] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:38:58 (1789051138) [ 2584.083186] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:39:05 (1789051145) [ 2592.453860] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:39:13 (1789051153) [ 2600.980913] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:39:21 (1789051161) [ 2612.189308] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:39:32 (1789051172) [ 2621.174918] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:39:41 (1789051181) [ 2628.152754] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:39:49 (1789051189) [ 2636.468551] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:39:57 (1789051197) [ 2645.047827] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:40:06 (1789051206) [ 2652.949697] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:40:13 (1789051213) [ 2660.748793] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:40:21 (1789051221) [ 2669.290403] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:40:30 (1789051230) [ 2676.925515] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:40:37 (1789051237) [ 2684.214581] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:40:45 (1789051245) [ 2692.263555] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:40:53 (1789051253) [ 2701.062803] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:41:01 (1789051261) [ 2709.311533] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:41:10 (1789051270) [ 2717.387884] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:41:18 (1789051278) [ 2717.550290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2717.587212] LNet: 137531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2717.604543] LNetError: 137531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2717.623303] LNet: Removed LNI 192.168.202.14@tcp2 [ 2717.707418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2717.886489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2717.918323] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2719.300791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2719.350099] LNet: 137680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.355341] LNetError: 137680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.367051] LNet: Removed LNI 192.168.202.14@tcp [ 2719.394230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2719.471167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2719.494216] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2726.562580] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:41:27 (1789051287) [ 2734.270666] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:41:35 (1789051295) [ 2741.757969] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:41:43 (1789051303) [ 2750.585571] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:41:51 (1789051311) [ 2758.712594] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:41:59 (1789051319) [ 2767.330778] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:42:08 (1789051328) [ 2776.005702] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:42:17 (1789051337) [ 2783.647993] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:42:24 (1789051344) [ 2791.201942] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:42:32 (1789051352) [ 2799.252217] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:42:40 (1789051360) [ 2806.314501] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:42:47 (1789051367) [ 2814.252533] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:42:55 (1789051375) [ 2822.389244] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2563 sec ========== 10:43:03 (1789051383) [ 2822.709612] LNet: 145151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.715387] LNetError: 145151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.728471] LNet: Removed LNI 192.168.202.14@tcp [ 2823.541645] Key type .llcrypt unregistered [ 2823.544448] Key type ._llcrypt unregistered [ 2838.100560] Key type ._llcrypt registered [ 2838.113594] Key type .llcrypt registered [ 2838.646646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2838.664983] alg: No test for adler32 (adler32-zlib) [ 2839.918167] Lustre: Lustre: Build Version: 2.17.58_40_gb4686d2 [ 2840.145656] LNet: Added LNI 192.168.202.14@tcp [8/256/0/180] [ 2841.839242] Key type lgssc registered [ 2843.420902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2914.638249] Lustre: Mounted lustre-client - version 2.17.58_40_gb4686d2 [ 2920.055727] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2928.845158] Lustre: Unmounted lustre-client [ 2966.204945] Key type lgssc unregistered [ 2966.551026] LNet: 148110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2966.566470] LNetError: 148110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.601234] LNet: Removed LNI 192.168.202.14@tcp [ 2968.554262] Key type .llcrypt unregistered [ 2968.556366] Key type ._llcrypt unregistered