[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 713723171 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.007844] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009022] pid_max: default: 32768 minimum: 301 [ 0.010175] LSM: Security Framework initializing [ 0.011077] Yama: becoming mindful. [ 0.012062] SELinux: Initializing. [ 0.014031] *** VALIDATE selinux *** [ 0.025117] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031566] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.033133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034174] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036075] *** VALIDATE tmpfs *** [ 0.038232] *** VALIDATE proc *** [ 0.040108] *** VALIDATE cgroup *** [ 0.041018] *** VALIDATE cgroup2 *** [ 0.042353] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.043181] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045041] Spectre V2 : User space: Vulnerable [ 0.046012] Speculative Store Bypass: Vulnerable [ 0.049766] debug: unmapping init [mem 0xffffffffae059000-0xffffffffae060fff] [ 0.051264] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.052856] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.053029] ... version: 2 [ 0.054020] ... bit width: 48 [ 0.055017] ... generic registers: 4 [ 0.056015] ... value mask: 0000ffffffffffff [ 0.057012] ... max period: 00007fffffffffff [ 0.058013] ... fixed-purpose events: 3 [ 0.059009] ... event mask: 000000070000000f [ 0.060258] rcu: Hierarchical SRCU implementation. [ 0.062488] smp: Bringing up secondary CPUs ... [ 0.063528] x86: Booting SMP configuration: [ 0.064024] .... node #0, CPUs: #1 #2 #3 [ 0.074216] smp: Brought up 1 node, 4 CPUs [ 0.076028] smpboot: Max logical packages: 1 [ 0.077011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.112029] node 0 deferred pages initialised in 33ms [ 0.115007] devtmpfs: initialized [ 0.116217] x86/mm: Memory block size: 128MB [ 0.119015] gcov: version magic: 0x41383552 [ 0.121476] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122211] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.123348] pinctrl core: initialized pinctrl subsystem [ 0.124173] [ 0.124790] ************************************************************* [ 0.125015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126010] ** ** [ 0.127010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.128010] ** ** [ 0.129011] ** This means that this kernel is built to expose internal ** [ 0.130014] ** IOMMU data structures, which may compromise security on ** [ 0.131014] ** your system. ** [ 0.132010] ** ** [ 0.133009] ** If you see this message and you are not debugging the ** [ 0.134009] ** kernel, report this immediately to your vendor! ** [ 0.135010] ** ** [ 0.136010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137011] ************************************************************* [ 0.138680] NET: Registered protocol family 16 [ 0.139490] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.140058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.141064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.142433] cpuidle: using governor menu [ 0.143913] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.146416] PCI: Using configuration type 1 for base access [ 0.148150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.155342] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.156017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.159119] cryptd: max_cpu_qlen set to 1000 [ 0.161401] ACPI: Added _OSI(Module Device) [ 0.163018] ACPI: Added _OSI(Processor Device) [ 0.165012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.172895] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.179682] ACPI: Interpreter enabled [ 0.182065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183014] ACPI: Using IOAPIC for interrupt routing [ 0.186115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191707] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.207605] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.209065] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.212107] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215177] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.221000] acpiphp: Slot [2] registered [ 0.223138] acpiphp: Slot [5] registered [ 0.224159] acpiphp: Slot [6] registered [ 0.226185] acpiphp: Slot [3] registered [ 0.228102] acpiphp: Slot [4] registered [ 0.229154] acpiphp: Slot [7] registered [ 0.231102] acpiphp: Slot [8] registered [ 0.233096] acpiphp: Slot [9] registered [ 0.234140] acpiphp: Slot [10] registered [ 0.236111] acpiphp: Slot [11] registered [ 0.237112] acpiphp: Slot [12] registered [ 0.239094] acpiphp: Slot [13] registered [ 0.240101] acpiphp: Slot [14] registered [ 0.241285] acpiphp: Slot [15] registered [ 0.243096] acpiphp: Slot [16] registered [ 0.244223] acpiphp: Slot [17] registered [ 0.245098] acpiphp: Slot [18] registered [ 0.246000] acpiphp: Slot [19] registered [ 0.250174] acpiphp: Slot [20] registered [ 0.255144] acpiphp: Slot [21] registered [ 0.258190] acpiphp: Slot [22] registered [ 0.262197] acpiphp: Slot [23] registered [ 0.264338] acpiphp: Slot [24] registered [ 0.266173] acpiphp: Slot [25] registered [ 0.270162] acpiphp: Slot [26] registered [ 0.273345] acpiphp: Slot [27] registered [ 0.275805] acpiphp: Slot [28] registered [ 0.278368] acpiphp: Slot [29] registered [ 0.280116] acpiphp: Slot [30] registered [ 0.283323] acpiphp: Slot [31] registered [ 0.285116] PCI host bridge to bus 0000:00 [ 0.288023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.304024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310332] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314881] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.327000] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.337518] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.341019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.344027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.347019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.349567] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.353039] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358867] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363063] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.372925] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377080] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382132] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395080] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414074] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.426305] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.457016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475162] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.479390] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481314] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.484396] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.486223] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.491047] iommu: Default domain type: Passthrough [ 0.492620] SCSI subsystem initialized [ 0.493115] ACPI: bus type USB registered [ 0.494236] usbcore: registered new interface driver usbfs [ 0.496088] usbcore: registered new interface driver hub [ 0.498118] usbcore: registered new device driver usb [ 0.500288] pps_core: LinuxPPS API ver. 1 registered [ 0.503012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.506176] PTP clock support registered [ 0.508784] EDAC MC: Ver: 3.0.0 [ 0.509000] PCI: Using ACPI for IRQ routing [ 0.511288] NetLabel: Initializing [ 0.513008] NetLabel: domain hash size = 128 [ 0.515009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.517153] NetLabel: unlabeled traffic allowed by default [ 0.519801] vgaarb: loaded [ 0.522004] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.524010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.533746] clocksource: Switched to clocksource kvm-clock [ 0.674725] VFS: Disk quotas dquot_6.6.0 [ 0.676173] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.678194] *** VALIDATE ramfs *** [ 0.679239] *** VALIDATE hugetlbfs *** [ 0.680407] pnp: PnP ACPI init [ 0.682679] pnp: PnP ACPI: found 6 devices [ 0.699551] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.702210] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.703973] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.705907] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.708211] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.710617] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.714162] NET: Registered protocol family 2 [ 0.718471] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.726321] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.730821] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.738984] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.743869] TCP: Hash tables configured (established 65536 bind 65536) [ 0.746725] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.749439] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.751857] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.754502] NET: Registered protocol family 1 [ 0.756780] RPC: Registered named UNIX socket transport module. [ 0.758626] RPC: Registered udp transport module. [ 0.760239] RPC: Registered tcp transport module. [ 0.761573] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.763631] NET: Registered protocol family 44 [ 0.765125] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.766807] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.768644] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.770543] PCI: CLS 0 bytes, default 64 [ 0.771843] Unpacking initramfs... [ 3.162198] debug: unmapping init [mem 0xffff96d73cc64000-0xffff96d73ffcffff] [ 3.171183] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.176112] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.181405] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.474229] Initialise system trusted keyrings [ 4.476746] Key type blacklist registered [ 4.479490] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.493518] zbud: loaded [ 4.498368] *** VALIDATE nfs *** [ 4.500321] *** VALIDATE nfs4 *** [ 4.502389] pstore: using deflate compression [ 4.510912] Platform Keyring initialized [ 4.708946] NET: Registered protocol family 38 [ 4.711365] Key type asymmetric registered [ 4.712851] Asymmetric key parser 'x509' registered [ 4.717701] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.722149] io scheduler mq-deadline registered [ 4.724700] io scheduler kyber registered [ 4.731720] io scheduler bfq registered [ 4.735257] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.740559] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.748777] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.754666] ACPI: Power Button [PWRF] [ 4.765774] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.778495] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.801218] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.844045] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.876046] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.883574] Non-volatile memory driver v1.3 [ 4.886339] Linux agpgart interface v0.103 [ 4.929524] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 4.933918] vda: detected capacity change from 0 to 74698752 [ 4.954715] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.959676] vdb: detected capacity change from 0 to 1073741824 [ 4.979089] libphy: Fixed MDIO Bus: probed [ 4.990826] usbcore: registered new interface driver usbserial_generic [ 4.995361] usbserial: USB Serial support registered for generic [ 4.999651] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.012797] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.016637] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.021236] mousedev: PS/2 mouse device common for all mice [ 5.026745] rtc_cmos 00:05: RTC can wake from S4 [ 5.031106] rtc_cmos 00:05: registered as rtc0 [ 5.033706] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.038670] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.043459] intel_pstate: CPU model not supported [ 5.063543] hid: raw HID events driver (C) Jiri Kosina [ 5.067356] usbcore: registered new interface driver usbhid [ 5.075849] usbhid: USB HID core driver [ 5.082828] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.084761] drop_monitor: Initializing network drop monitor service [ 5.095966] Initializing XFRM netlink socket [ 5.099679] NET: Registered protocol family 10 [ 5.103157] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.108358] Segment Routing with IPv6 [ 5.110670] NET: Registered protocol family 17 [ 5.114242] mpls_gso: MPLS GSO support [ 5.120522] RAS: Correctable Errors collector initialized. [ 5.122723] AVX version of gcm_enc/dec engaged. [ 5.125478] AES CTR mode by8 optimization enabled [ 5.230576] sched_clock: Marking stable (5230495061, 0)->(6452348071, -1221853010) [ 5.240318] registered taskstats version 1 [ 5.246483] Loading compiled-in X.509 certificates [ 5.251765] zswap: loaded using pool lzo/zbud [ 5.295825] Key type big_key registered [ 5.320622] Key type encrypted registered [ 5.326710] ima: No TPM chip found, activating TPM-bypass! [ 5.331888] ima: Allocated hash algorithm: sha1 [ 5.337650] ima: No architecture policies found [ 5.343284] evm: Initialising EVM extended attributes: [ 5.349326] evm: security.selinux [ 5.356562] evm: security.ima [ 5.357753] evm: security.capability [ 5.361374] evm: HMAC attrs: 0x1 [ 5.367263] rtc_cmos 00:05: setting system clock to 2026-08-08 10:42:52 UTC (1786185772) [ 5.380382] debug: unmapping init [mem 0xffffffffaf003000-0xffffffffaf1fffff] [ 5.387872] debug: unmapping init [mem 0xffffffffadd82000-0xffffffffae058fff] [ 5.401218] Write protecting the kernel read-only data: 28672k [ 5.408375] debug: unmapping init [mem 0xffffffffac403000-0xffffffffac5fffff] [ 5.414430] debug: unmapping init [mem 0xffffffffacd14000-0xffffffffacdfffff] [ 5.497172] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.517727] systemd[1]: Detected virtualization kvm. [ 5.522438] systemd[1]: Detected architecture x86-64. [ 5.527182] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.568433] systemd[1]: No hostname configured. [ 5.574112] systemd[1]: Set hostname to . [ 5.576791] random: systemd: uninitialized urandom read (16 bytes read) [ 5.582743] systemd[1]: Initializing machine ID from random generator. [ 5.799495] random: ln: uninitialized urandom read (6 bytes read) [ 6.019819] random: systemd: uninitialized urandom read (16 bytes read) [ 6.023206] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.029174] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 6.043036] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.918265] device-mapper: uevent: version 1.0.3 [ 7.922826] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.603198] virtio_net virtio0 ens2: renamed from eth0 [ 10.638246] random: fast init done [ 10.828884] scsi host0: ata_piix [ 10.865744] scsi host1: ata_piix [ 10.868622] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 10.873024] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.880129] random: crng init done [ 15.881716] random: 7 urandom warning(s) missed due to ratelimiting [ 19.065584] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 21.066740] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.102439] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.863348] SELinux: Disabled at runtime. [ 25.000272] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 25.026584] systemd[1]: Detected virtualization kvm. [ 25.034627] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.718283] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.724746] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.735195] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.738887] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.743237] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.753139] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.762293] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue Fil[ 27.280416] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 28.539404] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.796417] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 30.046955] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.193162] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.331448] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 34.987112] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.947549] NFS: Registering the id_resolver key type [ 35.959093] Key type id_resolver registered [ 35.964395] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started OpenSSH server daemon. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg417-client login: [ 104.380697] libcfs: loading out-of-tree module taints kernel. [ 104.586153] Key type ._llcrypt registered [ 104.595861] Key type .llcrypt registered [ 105.761658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 105.778541] alg: No test for adler32 (adler32-zlib) [ 107.212805] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 107.978596] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 109.671565] Key type lgssc registered [ 111.355549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.270007] hrtimer: interrupt took 3037483 ns [ 259.575517] Lustre: Mounted lustre-client - version 2.17.56_50_g336d872 [ 264.846622] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.536966] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing check_logdir /tmp/testlogs/ [ 284.779749] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing yml_node [ 285.153831] Lustre: lustre-OST0000-osc-ffff96d790847000: disconnect after 23s idle [ 289.393952] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 292.050880] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 295.100987] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 296.758432] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 8 06:47:42 EDT 2026 [ 314.928774] Lustre: Unmounted lustre-client [ 353.561272] Key type lgssc unregistered [ 353.967959] LNet: 8415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.973233] LNetError: 8415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 354.003465] LNet: Removed LNI 192.168.204.17@tcp [ 354.753539] Key type .llcrypt unregistered [ 354.756684] Key type ._llcrypt unregistered [ 382.433573] Key type ._llcrypt registered [ 382.436496] Key type .llcrypt registered [ 382.600389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 382.612617] alg: No test for adler32 (adler32-zlib) [ 383.519772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 383.662664] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 383.732210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 388.829790] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 388.989671] LNet: 9295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.013292] LNetError: 9295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.054324] LNet: Removed LNI 192.168.204.17@tcp [ 389.885553] Key type .llcrypt unregistered [ 389.889276] Key type ._llcrypt unregistered [ 399.945430] Key type ._llcrypt registered [ 399.950316] Key type .llcrypt registered [ 401.814775] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:49:26 (1786186166) [ 402.966631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 402.986723] alg: No test for adler32 (adler32-zlib) [ 404.088968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.279132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.310185] LNet: 9720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.348713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.441586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.474734] LNet: 9727:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 412.957978] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:49:38 (1786186178) [ 413.257884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.307741] LNet: 10307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.330349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.395341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 421.797905] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:49:47 (1786186187) [ 422.276782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.344665] LNet: 10896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.388369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.498193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 429.610619] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:49:55 (1786186195) [ 429.832035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.881946] LNet: 11476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.953685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.149731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 438.267438] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:50:03 (1786186203) [ 438.387901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.432378] LNet: 12060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.455614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.620066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 439.158147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 447.537995] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:50:12 (1786186212) [ 447.631194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.679505] LNet: 12649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.737734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.872467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 448.218160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 448.741084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 458.521887] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:50:23 (1786186223) [ 458.784876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.828446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.894231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 466.864693] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:50:32 (1786186232) [ 467.059476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.131260] LNet: 13829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.139282] LNet: 13829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 467.215649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.389739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 467.421598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 467.724511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 467.756852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 468.172625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 468.215444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 468.558551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 468.606526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 469.020820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 469.097165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 469.397892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 469.433160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 469.737590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 469.770727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 470.105702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 470.140454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 470.467786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 470.515882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 471.186073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 471.253360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 471.711870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 471.759729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 472.144499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 472.180468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 472.453202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 472.516517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 481.356479] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:50:46 (1786186246) [ 481.444638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.472843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.565313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 481.595364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 489.633882] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:50:55 (1786186255) [ 489.772444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.832128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.081395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 490.166792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 500.071788] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:51:05 (1786186265) [ 500.416057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.489071] LNet: 15692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.493954] LNet: 15692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 500.523906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.596813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 500.629230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 509.174011] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:51:14 (1786186274) [ 509.405632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.552363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.703994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 509.746545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 519.308563] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:51:24 (1786186284) [ 519.485713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.576227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.775991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 519.829795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 529.157573] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:51:34 (1786186294) [ 529.312834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.346100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.467151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 529.578541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 538.120362] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:51:43 (1786186303) [ 538.295357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.402539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.508684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.029279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.369692] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:51:52 (1786186312) [ 547.679844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.840550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.183633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.494491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.784750] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:52:03 (1786186323) [ 557.869827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.934910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.092120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 558.262396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 566.574769] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:52:11 (1786186331) [ 566.774587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.837022] LNet: 19863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.840168] LNet: 19863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 566.913986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.974719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.404766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 578.341730] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:52:22 (1786186342) [ 578.489859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.537939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.678432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 578.780637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 587.195528] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:52:32 (1786186352) [ 587.322710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.400223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.553516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.617934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 596.592344] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:52:41 (1786186361) [ 596.869750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.923942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.024870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 606.157478] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:52:51 (1786186371) [ 606.359710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.461903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.668852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 615.484410] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:53:01 (1786186381) [ 615.658660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.768902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.902378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 616.194644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 624.523615] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:53:09 (1786186389) [ 624.874579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.929244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.044658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 625.072803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 632.585518] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:53:17 (1786186397) [ 632.763162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.840246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.961383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 633.016907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 642.084223] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:53:27 (1786186407) [ 642.364054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.497036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.646337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 642.678892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 650.297520] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:53:36 (1786186416) [ 650.746484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.839742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.983985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 651.039686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 659.384410] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:53:45 (1786186425) [ 659.625609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.760509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.872732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 659.960364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 659.989970] LNetError: 25842:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 660.048589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 660.086932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 668.499581] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:53:53 (1786186433) [ 668.678519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.775724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.877861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XaFPQCDxnJ [ 668.958838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XaFPQCDxnJ [ 669.090309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XaFPQCDxnJ [ 678.252463] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:54:03 (1786186443) [ 678.416500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.502661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.661548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 678.692505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 678.738291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 687.117493] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 06:54:12 (1786186452) [ 687.337702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.506383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.726370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.782122] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 689.965904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 692.335639] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 694.559385] Key type lgssc registered [ 696.165891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 702.000603] LNetError: 27605:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.17@tcp99 has no available nets [ 705.671777] Key type lgssc unregistered [ 706.029792] LNet: 28716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.034763] LNet: 28716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 706.043237] LNetError: 28716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.060365] LNet: Removed LNI 192.168.204.17@tcp [ 706.885356] Key type .llcrypt unregistered [ 706.890079] Key type ._llcrypt unregistered [ 716.523029] Key type ._llcrypt registered [ 716.530118] Key type .llcrypt registered [ 718.037790] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:54:43 (1786186483) [ 718.894772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.906815] alg: No test for adler32 (adler32-zlib) [ 720.048670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.179768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 722.456786] LNet: Message forwarding will be enabled in 2s [ 725.535083] LNet: Message forwarding enabled [ 726.232851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 726.265969] LNet: Message forwarding will stop in 2s to 6s [ 729.898546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 732.523231] LNet: Message forwarding disabled [ 733.538963] LNet: Message forwarding will be enabled in 2s [ 737.632063] LNet: Message forwarding enabled [ 745.461380] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:55:10 (1786186510) [ 745.719679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.787189] LNet: 30117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.005261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.185311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 755.607976] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:55:20 (1786186520) [ 755.857550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.952372] LNet: 30696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.008711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.201516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 756.222595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 756.266970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 756.299966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 756.331484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 756.356200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 756.386330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 756.420503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 756.462447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 756.529694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 756.584257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 756.637671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 756.673458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 756.714496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 756.764684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 756.802583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 756.849105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 756.909724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 756.940578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 756.958726] LNetError: 30744:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 766.070657] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:55:30 (1786186530) [ 766.277959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.333066] LNet: 31312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.406608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.664792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 775.209488] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:55:40 (1786186540) [ 777.589479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg417-server [ 780.200788] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:55:45 (1786186545) [ 780.363479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.407498] LNet: 32243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.439483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.547857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.584184] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 781.327402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[18-22]@tcp [ 781.361316] LNet: 32382:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 781.403262] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 781.424335] LNetError: 32252:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.19@tcp has gone from up to down [ 782.449912] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 782.468522] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 783.483854] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 783.520096] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 785.528858] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 785.539901] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 789.616236] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.21@tcp is being used as a gateway but routing feature is not turned on [ 789.648486] LNetError: 32252:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 790.675924] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:55:56 (1786186556) [ 792.835318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg417-server [ 794.712987] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:56:00 (1786186560) [ 794.957800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.074882] LNet: 33311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.100096] LNetError: 33311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.134517] LNet: Removed LNI 192.168.204.17@tcp [ 795.200241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.244481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.265791] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 795.860223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[18-23/2]@tcp [ 795.877364] LNet: 33435:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 795.883690] LNet: 33435:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 795.902621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[18-23/2]@tcp [ 795.922816] LNetError: 33322:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.18@tcp has gone from up to down [ 795.933124] LNetError: 33322:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 804.553994] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:56:09 (1786186569) [ 804.921603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.964561] LNet: 34028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.975838] LNetError: 34028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.005385] LNet: Removed LNI 192.168.204.17@tcp [ 805.044676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.193637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 805.264476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.299723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.352344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 805.426475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.500921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.575637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 805.698852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.742467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.826542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 805.946332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.004114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.069695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 806.150171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.195815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.258637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 806.351595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.382975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.426757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 806.501275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.539827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.619663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 806.722273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.764881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.828860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 806.933758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.043698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.173819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 807.349956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.481344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.608211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 807.751857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.842493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.971813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 817.192930] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:56:22 (1786186582) [ 819.407355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg417-server [ 821.791353] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:56:27 (1786186587) [ 824.585502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg417-server [ 826.531777] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:56:32 (1786186592) [ 826.882757] LNet: 35428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.888962] LNet: 35428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 827.663634] Key type .llcrypt unregistered [ 827.669474] Key type ._llcrypt unregistered [ 836.924503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.000883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.005389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.009352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.565655] Key type ._llcrypt registered [ 839.567891] Key type .llcrypt registered [ 839.733872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.755832] alg: No test for adler32 (adler32-zlib) [ 840.728290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.869211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.926845] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 841.881591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 841.934480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 844.990079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 845.029969] LNet: Removed LNI 10.1.2.3@tcp [ 845.204124] LNet: 36155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.214445] LNetError: 36155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.231567] LNet: Removed LNI 192.168.204.17@tcp [ 846.204676] Key type .llcrypt unregistered [ 846.214469] Key type ._llcrypt unregistered [ 864.837955] Key type ._llcrypt registered [ 864.841109] Key type .llcrypt registered [ 866.228082] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:57:11 (1786186631) [ 867.280297] Key type .llcrypt unregistered [ 867.287479] Key type ._llcrypt unregistered [ 876.738959] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.856857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.866372] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.870632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.504815] Key type ._llcrypt registered [ 878.506927] Key type .llcrypt registered [ 878.756463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.786540] alg: No test for adler32 (adler32-zlib) [ 879.664861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.716422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.735187] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 880.384697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 880.414538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.465662] LNet: 37653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.472920] LNetError: 37653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.477974] LNet: Removed LNI 10.1.2.3@tcp [ 883.502723] LNet: Removed LNI 192.168.204.17@tcp [ 884.559737] Key type .llcrypt unregistered [ 884.564837] Key type ._llcrypt unregistered [ 902.418864] Key type ._llcrypt registered [ 902.420340] Key type .llcrypt registered [ 904.357902] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:57:49 (1786186669) [ 905.316491] Key type .llcrypt unregistered [ 905.319319] Key type ._llcrypt unregistered [ 914.556173] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 914.976473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 915.001856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 915.016458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.097919] Key type ._llcrypt registered [ 918.107505] Key type .llcrypt registered [ 918.575251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.613173] alg: No test for adler32 (adler32-zlib) [ 919.535666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.596588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 919.619963] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 922.440378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 922.456483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.373935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 925.440043] LNet: Removed LNI 10.1.2.3@tcp [ 925.491794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 925.569822] LNet: Removed LNI 10.1.2.31@tcp [ 925.590755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 925.666331] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.690596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 926.714157] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 927.884994] LNet: 39580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.906827] LNetError: 39580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 927.922515] LNet: Removed LNI 10.1.2.31@tcp [ 927.942772] LNet: Removed LNI 10.1.2.3@tcp [ 928.657617] Key type .llcrypt unregistered [ 928.662735] Key type ._llcrypt unregistered [ 947.895422] Key type ._llcrypt registered [ 947.896899] Key type .llcrypt registered [ 949.757591] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:58:34 (1786186714) [ 950.506358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 950.540986] alg: No test for adler32 (adler32-zlib) [ 951.555388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.643574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 951.660557] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 953.204331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.262934] LNet: 40539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.277115] LNetError: 40539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.299943] LNet: Removed LNI 192.168.204.17@tcp [ 953.335567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.446458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.17@tcp [ 953.499706] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 955.047338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.17@tcp [ 955.096058] LNet: Removed LNI 192.168.204.17@tcp [ 955.274589] LNet: 40675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.064703] Key type .llcrypt unregistered [ 956.068720] Key type ._llcrypt unregistered [ 971.734296] Key type ._llcrypt registered [ 971.740459] Key type .llcrypt registered [ 973.532502] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:58:58 (1786186738) [ 976.011878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg417-server [ 977.909773] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:59:03 (1786186743) [ 978.762766] Key type .llcrypt unregistered [ 978.766680] Key type ._llcrypt unregistered [ 985.426241] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 985.897235] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 985.965524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 985.987666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 987.729703] Key type ._llcrypt registered [ 987.732274] Key type .llcrypt registered [ 987.875139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.886843] alg: No test for adler32 (adler32-zlib) [ 988.798658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.849320] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 989.528586] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 992.018518] LNet: 42512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.030440] LNetError: 42512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.041691] LNet: Removed LNI 192.168.204.17@tcp1 [ 992.053089] LNet: Removed LNI 10.1.2.3@tcp2 [ 992.939175] Key type .llcrypt unregistered [ 992.941566] Key type ._llcrypt unregistered [ 1007.968266] Key type ._llcrypt registered [ 1007.973611] Key type .llcrypt registered [ 1009.602816] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:59:34 (1786186774) [ 1010.379894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1010.392711] alg: No test for adler32 (adler32-zlib) [ 1011.315205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.360898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.390935] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1012.322920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.344750] LNet: 43487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.351789] LNetError: 43487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.362608] LNet: Removed LNI 192.168.204.17@tcp [ 1012.378471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.457878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1012.491924] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1023.285694] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:59:48 (1786186788) [ 1023.661628] LNet: 44072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.687451] LNetError: 44072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.718305] LNet: Removed LNI 192.168.204.17@tcp [ 1025.151507] Key type .llcrypt unregistered [ 1025.155619] Key type ._llcrypt unregistered [ 1036.302887] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1036.622560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1036.639723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1036.672658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.148763] Key type ._llcrypt registered [ 1040.153895] Key type .llcrypt registered [ 1040.472432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.504702] alg: No test for adler32 (adler32-zlib) [ 1041.418361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.500307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1041.557088] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1042.548453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1042.564989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.171349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.230556] LNet: 44829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.256969] LNetError: 44829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.274934] LNet: Removed LNI 10.1.2.3@tcp [ 1046.292433] LNet: Removed LNI 192.168.204.17@tcp [ 1046.339363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.471628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1046.497487] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1046.501668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1047.267323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.339605] LNet: 44856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.355931] LNetError: 44856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.377379] LNetError: 44856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1047.400099] LNet: Removed LNI 10.1.2.3@tcp [ 1047.418344] LNet: Removed LNI 192.168.204.17@tcp [ 1047.452238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.595485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1047.631066] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1047.642904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.345550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1048.395902] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 1050.672149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.728250] LNet: 45019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.744442] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.761877] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1050.782931] LNet: Removed LNI 10.1.2.3@tcp [ 1050.787777] LNet: Removed LNI 192.168.204.17@tcp [ 1050.813103] LNet: Removed LNI 192.168.204.17@tcp2 [ 1050.879857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.054222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1051.148308] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1051.162396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1051.191963] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 1063.349381] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:00:28 (1786186828) [ 1063.531146] LNet: 45610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.551107] LNetError: 45610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.564383] LNetError: 45610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1063.583525] LNet: Removed LNI 10.1.2.3@tcp [ 1063.587302] LNet: Removed LNI 192.168.204.17@tcp [ 1063.637707] LNet: Removed LNI 192.168.204.17@tcp2 [ 1064.572592] Key type .llcrypt unregistered [ 1064.576336] Key type ._llcrypt unregistered [ 1074.135494] Key type ._llcrypt registered [ 1074.140440] Key type .llcrypt registered [ 1074.456555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.499573] alg: No test for adler32 (adler32-zlib) [ 1075.688306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.798417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.834913] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1077.643623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.664844] LNet: 46018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.676665] LNetError: 46018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.688702] LNet: Removed LNI 192.168.204.17@tcp [ 1077.709708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.781069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1077.806658] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1078.207896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.232868] LNet: 46048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.238732] LNetError: 46048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.253940] LNet: Removed LNI 192.168.204.17@tcp [ 1078.283531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.366760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1078.404281] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1079.242952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1079.286967] LNet: 46078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.294658] LNetError: 46078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.313448] LNet: Removed LNI 192.168.204.17@tcp [ 1079.329917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.393930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1079.428436] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1080.056961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1080.336737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1080.341644] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1080.367454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1083.341490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.387546] LNet: 46158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.416630] LNetError: 46158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.450800] LNet: Removed LNI 192.168.204.17@tcp [ 1083.520823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.722888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1083.789307] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1094.874510] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:00:59 (1786186859) [ 1095.181587] LNet: 46763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.207223] LNetError: 46763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.275625] LNet: Removed LNI 192.168.204.17@tcp [ 1096.558624] Key type .llcrypt unregistered [ 1096.563945] Key type ._llcrypt unregistered [ 1107.380511] Key type ._llcrypt registered [ 1107.383590] Key type .llcrypt registered [ 1107.644915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.662676] alg: No test for adler32 (adler32-zlib) [ 1108.728113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.802080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1108.832405] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1109.655916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1109.671638] LNet: 47174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.675741] LNetError: 47174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.685501] LNet: Removed LNI 192.168.204.17@tcp [ 1109.721267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.816954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1109.879382] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1119.291101] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:01:24 (1786186884) [ 1119.846567] LNet: 47767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.875427] LNetError: 47767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.911096] LNet: Removed LNI 192.168.204.17@tcp [ 1121.223763] Key type .llcrypt unregistered [ 1121.227843] Key type ._llcrypt unregistered [ 1131.370445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1131.800419] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1131.842979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1131.850208] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1134.742406] Key type ._llcrypt registered [ 1134.745462] Key type .llcrypt registered [ 1135.058201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.098386] alg: No test for adler32 (adler32-zlib) [ 1136.207737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.302770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1136.339321] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1137.777511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1137.829637] LNet: 48226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.846548] LNetError: 48226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.882841] LNet: Removed LNI 192.168.204.17@tcp [ 1137.924182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.050539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1138.083189] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1148.518201] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:01:54 (1786186914) [ 1148.710779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1148.769103] LNet: 48824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.820776] LNetError: 48824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.851234] LNet: Removed LNI 192.168.204.17@tcp [ 1148.871859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1148.967497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1157.733416] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:02:02 (1786186922) [ 1158.206971] LNet: 49408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.212567] Key type .llcrypt unregistered [ 1159.217195] Key type ._llcrypt unregistered [ 1167.018988] Key type ._llcrypt registered [ 1167.028415] Key type .llcrypt registered [ 1167.399032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.429961] alg: No test for adler32 (adler32-zlib) [ 1168.541244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.669324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1168.732642] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1169.909774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1169.931784] LNet: 49810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.936760] LNetError: 49810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.950171] LNet: Removed LNI 192.168.204.17@tcp [ 1169.968582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.027286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1170.046050] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1177.373755] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:02:22 (1786186942) [ 1177.598762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1177.694708] LNet: 50397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.706267] LNetError: 50397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.755228] LNet: Removed LNI 192.168.204.17@tcp [ 1177.843062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.042897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1186.584588] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:02:32 (1786186952) [ 1189.304657] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1191.786550] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:02:37 (1786186957) [ 1194.123986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1196.213230] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:02:41 (1786186961) [ 1198.378871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1200.247621] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:02:45 (1786186965) [ 1202.390757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1204.435557] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:02:49 (1786186969) [ 1207.109405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1209.751312] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:02:54 (1786186974) [ 1209.929506] LNet: 52450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.347513] Key type .llcrypt unregistered [ 1211.351974] Key type ._llcrypt unregistered [ 1221.362857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1221.748095] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1221.757412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1221.766080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1224.779417] Key type ._llcrypt registered [ 1224.782046] Key type .llcrypt registered [ 1225.050938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1225.070844] alg: No test for adler32 (adler32-zlib) [ 1226.083178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.258115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1226.317056] LNet: Added LNI 192.168.204.17@tcp [8/512/0/180] [ 1226.332350] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1226.346900] LNet: 52784:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1226.584506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.618710] LNet: 52803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.630790] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1226.640746] LNet: Removed LNI 10.1.2.3@tcp [ 1226.644989] LNetError: 52803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.660713] LNet: Removed LNI 192.168.204.17@tcp [ 1226.684293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.793915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1226.824429] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1226.935959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.973846] LNet: 52827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.993334] LNetError: 52827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.007987] LNet: Removed LNI 192.168.204.17@tcp [ 1227.049547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.153845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1227.178385] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1227.186296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1227.190534] LNet: 52835:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1227.335912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.353684] LNet: 52853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.366715] LNetError: 52853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1227.378284] LNet: Removed LNI 10.1.2.3@tcp [ 1227.383519] LNetError: 52853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.400678] LNet: Removed LNI 192.168.204.17@tcp [ 1227.423380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.490704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1227.511052] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1227.520892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1227.526220] LNet: 52861:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1227.845569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.902278] LNet: 52887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.914496] LNetError: 52887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1227.920345] LNet: Removed LNI 10.1.2.3@tcp [ 1227.922487] LNetError: 52887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.951387] LNet: Removed LNI 192.168.204.17@tcp [ 1227.986800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.139813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1228.188550] LNet: Added LNI 192.168.204.17@tcp [8/256/0/182] [ 1228.199874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1228.325280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.375362] LNet: 52915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.381370] LNetError: 52915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1228.390435] LNet: Removed LNI 10.1.2.3@tcp [ 1228.393580] LNetError: 52915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.407453] LNet: Removed LNI 192.168.204.17@tcp [ 1228.446845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.563524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1228.587631] LNet: Added LNI 192.168.204.17@tcp [8/256/0/182] [ 1228.591361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1228.593321] LNet: 52923:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1228.656286] LNet: Removed LNI 10.1.2.3@tcp [ 1228.666977] LNet: Removed LNI 192.168.204.17@tcp [ 1229.528617] Key type .llcrypt unregistered [ 1229.530287] Key type ._llcrypt unregistered [ 1247.859601] Key type ._llcrypt registered [ 1247.861822] Key type .llcrypt registered [ 1249.459525] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:03:34 (1786187014) [ 1251.806849] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1254.298769] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:03:39 (1786187019) [ 1254.838850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.861771] alg: No test for adler32 (adler32-zlib) [ 1255.758178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.801642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1255.820184] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1256.522771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.17 [ 1256.572524] LNet: 54193:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1256.582116] LNet: Removed LNI 192.168.204.17@tcp [ 1265.065602] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:03:50 (1786187030) [ 1265.266162] LNet: 54756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.280159] LNetError: 54756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.317788] LNet: Removed LNI 192.168.204.17@tcp [ 1266.234474] Key type .llcrypt unregistered [ 1266.237705] Key type ._llcrypt unregistered [ 1274.377198] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1274.721451] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1277.667407] Key type ._llcrypt registered [ 1277.670981] Key type .llcrypt registered [ 1278.025358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.065417] alg: No test for adler32 (adler32-zlib) [ 1279.346930] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1288.632447] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:04:13 (1786187053) [ 1288.791763] LNet: 55675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.808620] LNetError: 55675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1288.832533] LNet: Removed LNI 10.1.2.3@tcp [ 1289.620696] Key type .llcrypt unregistered [ 1289.624735] Key type ._llcrypt unregistered [ 1298.477498] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1298.921776] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1298.927894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1298.933945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1301.854054] Key type ._llcrypt registered [ 1301.857247] Key type .llcrypt registered [ 1302.163510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1302.196448] alg: No test for adler32 (adler32-zlib) [ 1303.159739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1310.607330] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:04:35 (1786187075) [ 1312.290627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1314.511289] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:04:39 (1786187079) [ 1314.904391] LNet: 56873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.915777] LNetError: 56873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1314.948716] LNet: Removed LNI 10.1.2.3@tcp [ 1315.856856] Key type .llcrypt unregistered [ 1315.859684] Key type ._llcrypt unregistered [ 1323.748104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1324.082991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1324.087497] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1324.094176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1325.874327] Key type ._llcrypt registered [ 1325.880085] Key type .llcrypt registered [ 1326.234361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.249996] alg: No test for adler32 (adler32-zlib) [ 1327.197477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.348742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1337.216960] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:05:03 (1786187103) [ 1337.863789] LNet: 58100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.868861] LNetError: 58100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1337.895157] LNet: Removed LNI 10.1.2.3@tcp [ 1338.687498] Key type .llcrypt unregistered [ 1338.693227] Key type ._llcrypt unregistered [ 1345.166419] Key type ._llcrypt registered [ 1345.172564] Key type .llcrypt registered [ 1345.392064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1345.414352] alg: No test for adler32 (adler32-zlib) [ 1346.299924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1346.327750] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1354.786903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1356.114499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_interrupt -r 1 [ 1356.206215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_interrupt -r 1 [ 1356.298164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1356.345488] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1356.576396] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1356.800921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1356.872321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1356.935254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.971275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.007486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1357.066359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.141758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.233932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.285739] LNet: 58542:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.320668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.382342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.444031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.520955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.608660] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1357.620436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.700556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1357.768626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.686957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_dropped -r 1 [ 1358.737897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_dropped -r 1 [ 1358.792205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1358.820417] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1359.259508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1359.325939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1359.406905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1359.462150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1359.515902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1359.597158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1359.649966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1359.718611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.755167] LNet: 58652:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1359.786677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.856776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.924719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.988259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.059381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.099442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1360.148902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.279363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_aborted -r 1 [ 1361.331750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_aborted -r 1 [ 1361.386212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1361.421317] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1361.855922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.928798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1361.982456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1362.042355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1362.067420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.095723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.142723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.196378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.225807] LNet: 58761:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.272818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.328926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.363599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.391268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.434189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.467600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1362.506211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1363.159983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_no_route -r 1 [ 1363.213439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_no_route -r 1 [ 1363.273475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1363.501829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.523283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.549173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.571084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.598265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.620906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.650928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.673450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.697307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.723942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1363.754654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1363.784642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1363.812926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1363.842791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1363.887873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1364.627990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_timeout -r 1 [ 1364.673182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_timeout -r 1 [ 1364.745490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1365.167890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.202072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1365.227821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.249484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1365.271697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1365.296453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1365.328928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1365.359805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1365.383903] LNet: 58980:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.390631] LNet: 58980:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1365.403741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1365.434461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1365.463959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1365.499692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.535869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.565384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.600148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1366.354983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_error -r 1 [ 1366.429238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_error -r 1 [ 1366.473304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1366.500223] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1366.506650] LNet: Skipped 2 previous similar messages [ 1366.831508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1366.870684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1366.899393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1366.931662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1366.960734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1366.991341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.017899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.045472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.082341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.136494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.170763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.196802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.226581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.296844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.363618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1367.661330] LNet: 59134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.668475] LNetError: 59134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.682411] LNet: Removed LNI 192.168.204.17@tcp [ 1368.050607] Key type .llcrypt unregistered [ 1368.057106] Key type ._llcrypt unregistered [ 1388.690618] Key type ._llcrypt registered [ 1388.692651] Key type .llcrypt registered [ 1390.038741] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:05:55 (1786187155) [ 1391.224347] Key type .llcrypt unregistered [ 1391.227743] Key type ._llcrypt unregistered [ 1397.676654] Key type ._llcrypt registered [ 1397.678979] Key type .llcrypt registered [ 1397.852484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1397.872970] alg: No test for adler32 (adler32-zlib) [ 1398.747532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1398.774748] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1406.379212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1418.313554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.376730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.410041] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1420.069673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_interrupt -r 1 [ 1420.117707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_interrupt -r 1 [ 1420.160117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_interrupt -r 1 [ 1420.195822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_interrupt -r 1 [ 1420.238405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_interrupt -r 1 [ 1420.269491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_interrupt -r 1 [ 1420.293891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_interrupt -r 1 [ 1420.324722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_interrupt -r 1 [ 1420.349791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1420.367946] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1420.375792] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1420.558631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1420.581950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1420.608281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1420.633629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1420.658987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1420.683485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1420.709709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1420.733919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1420.755494] LNet: 60608:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1420.783455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1420.832378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1420.878962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1420.926701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.993854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.065545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1421.157821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1421.922224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_dropped -r 1 [ 1421.974432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_dropped -r 1 [ 1422.015696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_dropped -r 1 [ 1422.063426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_dropped -r 1 [ 1422.119694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_dropped -r 1 [ 1422.174866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_dropped -r 1 [ 1422.218524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_dropped -r 1 [ 1422.280823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_dropped -r 1 [ 1422.341773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1422.363903] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1422.369935] LNet: Skipped 2 previous similar messages [ 1422.823149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1422.887435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1422.946568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1423.006996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1423.053823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1423.099680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1423.153822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1423.214410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1423.242952] LNet: 60729:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1423.277554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1423.335310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1423.396074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.419536] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1423.447575] LNet: Skipped 3 previous similar messages [ 1423.472826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1423.527135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1423.585492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1423.632105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1424.480786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_aborted -r 1 [ 1424.513860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_aborted -r 1 [ 1424.565616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_aborted -r 1 [ 1424.627178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_aborted -r 1 [ 1424.679960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_aborted -r 1 [ 1424.756065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_aborted -r 1 [ 1424.810690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_aborted -r 1 [ 1424.895597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_aborted -r 1 [ 1424.987309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1425.185842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1425.209530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1425.235389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1425.259898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1425.285549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1425.308228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1425.334204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1425.360492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1425.385481] LNet: 60851:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1425.408363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1425.445857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1425.476646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1425.508422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1425.543405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1425.572421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1425.624260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1426.548517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_no_route -r 1 [ 1426.652955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_no_route -r 1 [ 1426.724556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_no_route -r 1 [ 1426.762598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_no_route -r 1 [ 1426.806199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_no_route -r 1 [ 1426.861300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_no_route -r 1 [ 1426.920447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_no_route -r 1 [ 1427.002682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_no_route -r 1 [ 1427.092177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1427.159197] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1427.176148] LNet: Skipped 3 previous similar messages [ 1427.615338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1427.650464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1427.678832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1427.712360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1427.740714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1427.770633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1427.797359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1427.822516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1427.840563] LNet: 60972:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1427.869574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1427.907319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1427.941350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1427.970330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1427.993549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1428.024965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1428.056277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.665549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_timeout -r 1 [ 1428.719523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_timeout -r 1 [ 1428.771632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_timeout -r 1 [ 1428.856984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_timeout -r 1 [ 1428.935054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_timeout -r 1 [ 1429.002141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_timeout -r 1 [ 1429.038313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_timeout -r 1 [ 1429.071311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_timeout -r 1 [ 1429.111505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1429.418648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1429.451207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1429.482528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1429.523477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1429.557480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1429.598461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1429.628407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1429.667522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1429.703968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1429.749232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1429.787695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1429.822576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1429.858492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.891775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1429.933520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.908805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_error -r 1 [ 1430.955764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_error -r 1 [ 1431.021128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_error -r 1 [ 1431.105191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_error -r 1 [ 1431.186505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_error -r 1 [ 1431.287653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_error -r 1 [ 1431.375250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_error -r 1 [ 1431.424948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_error -r 1 [ 1431.474774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1431.490696] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1431.496957] LNet: Skipped 5 previous similar messages [ 1431.863894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1431.909477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1431.948438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1431.996631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1432.036196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1432.076949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1432.132177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1432.164963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1432.192928] LNet: 61215:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.201118] LNet: 61215:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1432.247857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1432.323296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1432.415023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1432.482058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1432.581343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1432.657894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1432.749969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1438.670215] LNet: 61295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.676529] LNetError: 61295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.687313] LNet: Removed LNI 192.168.204.17@tcp [ 1439.718765] LNet: Removed LNI 192.168.204.17@tcp1 [ 1440.579708] Key type .llcrypt unregistered [ 1440.585346] Key type ._llcrypt unregistered [ 1460.871519] Key type ._llcrypt registered [ 1460.881134] Key type .llcrypt registered [ 1462.376205] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:07:07 (1786187227) [ 1463.699496] Key type .llcrypt unregistered [ 1463.704551] Key type ._llcrypt unregistered [ 1471.804298] Key type ._llcrypt registered [ 1471.808412] Key type .llcrypt registered [ 1472.134503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1472.160974] alg: No test for adler32 (adler32-zlib) [ 1473.128805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1473.193708] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1483.442974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1484.776288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_dropped -r 1 [ 1484.897985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_dropped -r 1 [ 1485.054943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1485.114332] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1485.680701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.742341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1485.775410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.805695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.876714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.953459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1486.058771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1486.124551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1486.190951] LNet: 62552:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1486.253800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1486.296513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1486.325683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1486.355918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1486.406805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1486.449419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.477188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1487.174608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_error -r 1 [ 1487.229701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_error -r 1 [ 1487.305252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1487.352784] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1487.773352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.838456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1487.874436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.903358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.946679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1487.989728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1488.046663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1488.093911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1488.126025] LNet: 62661:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1488.155331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1488.190791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1488.224619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.269837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.312610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.368907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.422762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1488.956145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_timeout -r 1 [ 1489.000698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_timeout -r 1 [ 1489.054147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1489.086350] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1489.329624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1489.379078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1489.424682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1489.467478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1489.504557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1489.572474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1489.627645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1489.703804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1489.724361] LNet: 62770:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1489.741992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1489.771927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1489.797639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1489.827643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1489.864204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1489.906819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1489.962976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1490.458396] LNet: 62816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.478718] LNetError: 62816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.497734] LNet: Removed LNI 192.168.204.17@tcp [ 1492.120586] Key type .llcrypt unregistered [ 1492.125052] Key type ._llcrypt unregistered [ 1519.444965] Key type ._llcrypt registered [ 1519.450312] Key type .llcrypt registered [ 1521.415851] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:08:06 (1786187286) [ 1522.278293] Key type .llcrypt unregistered [ 1522.283682] Key type ._llcrypt unregistered [ 1529.575082] Key type ._llcrypt registered [ 1529.577056] Key type .llcrypt registered [ 1529.930528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1529.950693] alg: No test for adler32 (adler32-zlib) [ 1530.850967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1530.881139] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1541.572392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1555.833518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.906909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.952373] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1558.869571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_dropped -r 1 [ 1558.923414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e remote_dropped -r 1 [ 1558.994850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_dropped -r 1 [ 1559.066057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e remote_dropped -r 1 [ 1559.162740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e remote_dropped -r 1 [ 1559.236228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e remote_dropped -r 1 [ 1559.291810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e remote_dropped -r 1 [ 1559.378553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e remote_dropped -r 1 [ 1559.439566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1559.476697] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1559.488110] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.117@tcp [ 1559.916340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1559.980516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1560.053958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1560.118919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1560.171365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1560.238983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1560.328709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1560.408771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1560.438375] LNet: 64292:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.506892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1560.547212] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1560.570270] LNet: Skipped 3 previous similar messages [ 1560.633401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1560.722507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1560.806949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.874926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.914526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1560.954721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1560.983927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1561.775893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_error -r 1 [ 1561.860934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e remote_error -r 1 [ 1561.932513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_error -r 1 [ 1562.010759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e remote_error -r 1 [ 1562.056771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e remote_error -r 1 [ 1562.139289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e remote_error -r 1 [ 1562.209962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e remote_error -r 1 [ 1562.259968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e remote_error -r 1 [ 1562.345927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1562.386884] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1562.844521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1562.925833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1563.009507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1563.132131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1563.210954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1563.268490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1563.335286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1563.394744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1563.439275] LNet: 64415:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1563.483847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1563.565728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1563.625793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1563.694721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1563.742883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1563.787103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1563.829727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1563.865510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1564.781217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_timeout -r 1 [ 1564.859132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e remote_timeout -r 1 [ 1564.939077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_timeout -r 1 [ 1565.013949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e remote_timeout -r 1 [ 1565.109548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e remote_timeout -r 1 [ 1565.186598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e remote_timeout -r 1 [ 1565.270612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e remote_timeout -r 1 [ 1565.330627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e remote_timeout -r 1 [ 1565.381989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.117@tcp [ 1565.414244] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1565.868368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1565.964936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1566.038562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1566.117861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1566.183872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1566.230899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1566.307927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1566.413130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1566.444210] LNet: 64539:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1566.492607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1566.572815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1566.624649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1566.655337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1566.707339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1566.770102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1566.835820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1566.888272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1574.939439] LNet: 64621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.947971] LNetError: 64621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.968487] LNet: Removed LNI 192.168.204.17@tcp [ 1575.977408] LNetError: 64621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.009877] LNet: Removed LNI 192.168.204.17@tcp1 [ 1576.887871] Key type .llcrypt unregistered [ 1576.895435] Key type ._llcrypt unregistered [ 1604.884258] Key type ._llcrypt registered [ 1604.886593] Key type .llcrypt registered [ 1606.658295] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:09:31 (1786187371) [ 1608.636394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1610.481624] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:09:35 (1786187375) [ 1611.425305] Key type .llcrypt unregistered [ 1611.430431] Key type ._llcrypt unregistered [ 1619.814855] Key type ._llcrypt registered [ 1619.824962] Key type .llcrypt registered [ 1620.136902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1620.171435] alg: No test for adler32 (adler32-zlib) [ 1621.312107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1621.410641] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1630.192578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1631.140934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e network_timeout -r 1 [ 1631.188105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e network_timeout -r 1 [ 1631.251980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1631.284678] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1631.507751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1631.569883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1631.625629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1631.712199] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1631.735518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1631.798804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1631.825403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1631.879305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1631.914500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1631.929367] LNet: 66173:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.951426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1631.979370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.008859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.052739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.082983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.119596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1632.490564] LNet: 66216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.496958] LNetError: 66216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.573714] LNet: Removed LNI 192.168.204.17@tcp [ 1633.954466] Key type .llcrypt unregistered [ 1633.956179] Key type ._llcrypt unregistered [ 1659.699719] Key type ._llcrypt registered [ 1659.702820] Key type .llcrypt registered [ 1660.091443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1660.130575] alg: No test for adler32 (adler32-zlib) [ 1661.136693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1661.179932] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1669.591430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1680.116815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.211173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1680.253462] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1682.429930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e network_timeout -r 1 [ 1682.475753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e network_timeout -r 1 [ 1682.509406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e network_timeout -r 1 [ 1682.539761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e network_timeout -r 1 [ 1682.573663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e network_timeout -r 1 [ 1682.601681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e network_timeout -r 1 [ 1682.637439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e network_timeout -r 1 [ 1682.670463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e network_timeout -r 1 [ 1682.712621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 1682.736437] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1682.956670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.995808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1683.036872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1683.072577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1683.109706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1683.152345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1683.201769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1683.237348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1683.277073] LNet: 67102:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1683.316163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1683.355549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1683.394740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1683.423845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1683.457668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1683.495545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1689.224186] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1689.825692] LNet: 67180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.837704] LNetError: 67180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.846991] LNet: Removed LNI 192.168.204.17@tcp [ 1689.862867] LNet: Removed LNI 192.168.204.17@tcp1 [ 1690.412594] Key type .llcrypt unregistered [ 1690.414206] Key type ._llcrypt unregistered [ 1714.986853] Key type ._llcrypt registered [ 1714.991618] Key type .llcrypt registered [ 1716.420854] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:11:21 (1786187481) [ 1716.944614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.960287] alg: No test for adler32 (adler32-zlib) [ 1717.907977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.020734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.050628] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1719.251364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1719.296846] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1720.591401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 1720.683721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1720.741855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1720.790723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1720.835132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1720.871727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1720.911038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.17@tcp [ 1720.931875] LNetError: 68280:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.17@tcp to recovery [ 1721.055310] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1721.058127] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1723.103393] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1723.722270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1727.200356] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1727.765986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1727.802329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1727.873393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.903666] LNet: 68377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.915921] LNetError: 68377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.929588] LNet: Removed LNI 192.168.204.17@tcp [ 1727.942523] LNet: Removed LNI 192.168.204.17@tcp1 [ 1727.960937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.010861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.035636] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1729.332149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1729.380858] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1730.352926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 1730.410980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1730.510941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1730.595213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1730.670421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1730.751605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1730.841677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1730.913907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.17@tcp [ 1730.953538] LNetError: 68650:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.17@tcp to recovery [ 1731.106085] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1733.811660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1737.247804] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1737.254687] LNet: Skipped 1 previous similar message [ 1742.520414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1742.582177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1742.679031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1751.918484] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:11:57 (1786187517) [ 1752.001749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.041104] LNet: 69353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.047092] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.057425] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1752.066153] LNet: Removed LNI 192.168.204.17@tcp [ 1752.082234] LNet: Removed LNI 192.168.204.17@tcp1 [ 1752.168210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.230841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1752.263673] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1752.972455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1753.001210] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1754.021752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 1754.195445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1754.269448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1754.337682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1754.388693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1754.439226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1754.490491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.17@tcp --health 0 [ 1754.698113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1755.295547] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.17@tcp [ 1755.295602] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1755.324260] LNet: Skipped 1 previous similar message [ 1769.752277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1770.170250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1770.237381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.275990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.17@tcp --health 500 [ 1770.383695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1777.347655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1777.515930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1777.578873] LNet: 69843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.589640] LNetError: 69843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.603264] LNetError: 69843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1777.609836] LNet: Removed LNI 192.168.204.17@tcp [ 1777.621208] LNet: Removed LNI 192.168.204.17@tcp1 [ 1777.667642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.769846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.812069] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1778.752772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1778.775310] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1780.408594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 1780.452817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1780.574373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1780.639408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1780.692677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1780.745847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1780.804614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1780.869893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.17@tcp --health 0 [ 1781.855868] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1781.861561] LNet: Skipped 3 previous similar messages [ 1782.367647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1792.123958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1792.215586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1792.285273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1801.468318] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:12:46 (1786187566) [ 1802.613284] LNet: 70877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.619436] LNetError: 70877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.626515] LNetError: 70877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1802.633433] LNet: Removed LNI 192.168.204.17@tcp [ 1802.651450] LNet: Removed LNI 192.168.204.17@tcp1 [ 1804.096349] Key type .llcrypt unregistered [ 1804.104822] Key type ._llcrypt unregistered [ 1813.709681] Key type ._llcrypt registered [ 1813.711561] Key type .llcrypt registered [ 1813.952981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1813.973641] alg: No test for adler32 (adler32-zlib) [ 1815.050317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1815.107045] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1828.005178] Lustre: DEBUG MARKER: Initial discovery [ 1828.126645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.117@tcp [ 1831.728788] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1832.089395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1832.154791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.117@tcp [ 1832.177832] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1832.210962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1832.254240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.117@tcp -r 1 -m GET -l 3 [ 1832.310596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.117@tcp -r 1 -m GET -e local_error [ 1832.368569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.117@tcp -r 1 -m PUT -l 6 [ 1832.554080] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1834.465252] Lustre: DEBUG MARKER: Force oleg417-client.virtnet to discover 192.168.204.117@tcp (in background) [ 1834.747225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.117@tcp [ 1836.994094] Lustre: DEBUG MARKER: Force oleg417-server to discover 192.168.204.17@tcp [ 1837.855372] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1840.520628] Lustre: DEBUG MARKER: Wait for 71645 [ 1841.378156] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 1842.878619] Lustre: DEBUG MARKER: Finished wait on 71645 [ 1842.966910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1843.069452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1843.598572] LNet: 72100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.607701] LNetError: 72100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.625658] LNet: Removed LNI 192.168.204.17@tcp [ 1844.125665] Key type .llcrypt unregistered [ 1844.127129] Key type ._llcrypt unregistered [ 1868.844760] Key type ._llcrypt registered [ 1868.847325] Key type .llcrypt registered [ 1870.218637] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:13:55 (1786187635) [ 1870.761603] Key type .llcrypt unregistered [ 1870.764784] Key type ._llcrypt unregistered [ 1876.529222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1876.675391] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1876.678627] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1876.683045] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1879.490132] Key type ._llcrypt registered [ 1879.492639] Key type .llcrypt registered [ 1879.770209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1879.778662] alg: No test for adler32 (adler32-zlib) [ 1880.654486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.694502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1880.713670] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1881.361938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1881.382052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1890.969279] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:14:15 (1786187655) [ 1891.075756] LNet: 74176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.096953] LNetError: 74176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1891.116235] LNet: Removed LNI 10.1.2.3@tcp [ 1891.138624] LNet: Removed LNI 192.168.204.17@tcp [ 1892.145391] Key type .llcrypt unregistered [ 1892.149068] Key type ._llcrypt unregistered [ 1900.902758] Key type ._llcrypt registered [ 1900.905935] Key type .llcrypt registered [ 1901.209652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1901.228616] alg: No test for adler32 (adler32-zlib) [ 1902.233603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.320652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.350714] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1903.310759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1903.336825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1904.887490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.17@tcp [ 1906.015166] LNetError: Unexpected error -22 connecting to 192.168.204.17@tcp at host 192.168.204.17:988 [ 1914.323605] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:14:39 (1786187679) [ 1914.636993] LNet: 75311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.654801] LNetError: 75311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1914.670836] LNet: Removed LNI 10.1.2.3@tcp [ 1914.723763] LNet: Removed LNI 192.168.204.17@tcp [ 1915.824808] Key type .llcrypt unregistered [ 1915.826625] Key type ._llcrypt unregistered [ 1927.094631] Key type ._llcrypt registered [ 1927.096024] Key type .llcrypt registered [ 1927.344672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1927.359898] alg: No test for adler32 (adler32-zlib) [ 1928.284825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.358774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1928.410075] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1929.792701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1929.844065] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 1931.239728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.17@tcp1 --nid 192.168.204.17@tcp2 [ 1936.442621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp1 192.168.204.17@tcp2 [ 1936.471175] LNetError: 76160:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.17@tcp1 to 192.168.204.17@tcp2 - Route table may be misconfigured [ 1936.496503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp2 192.168.204.17@tcp1 [ 1947.090097] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:15:12 (1786187712) [ 1947.229691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.280748] LNet: 76729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.290338] LNetError: 76729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.307827] LNet: Removed LNI 192.168.204.17@tcp1 [ 1948.326870] LNet: Removed LNI 192.168.204.17@tcp2 [ 1948.371141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.544463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.610911] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1949.816995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1949.838458] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1950.666116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 1950.716550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.17@tcp [ 1950.762398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp1 192.168.204.17@tcp1 [ 1950.869537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.17@tcp -d 192.168.204.17@tcp -e network_timeout [ 1950.952866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -e network_timeout [ 1951.050474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -e network_timeout [ 1951.127283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -e network_timeout [ 1951.190702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 1951.212744] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 1951.253839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1951.482100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1951.522838] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 1951.637578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1961.048124] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:15:26 (1786187726) [ 1961.159799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.193248] LNet: 77589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.217628] LNetError: 77589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.242687] LNetError: 77589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1961.274693] LNet: Removed LNI 192.168.204.17@tcp [ 1962.344028] LNet: Removed LNI 192.168.204.17@tcp1 [ 1962.395843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.617324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1963.124669] LNet: 77637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.738724] Key type .llcrypt unregistered [ 1963.743062] Key type ._llcrypt unregistered [ 1979.716752] Key type ._llcrypt registered [ 1979.720367] Key type .llcrypt registered [ 1981.419485] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:15:46 (1786187746) [ 1982.002187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1982.018865] alg: No test for adler32 (adler32-zlib) [ 1982.964277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.714816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1987.476523] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:15:52 (1786187752) [ 1987.675162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.767274] LNet: 78730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.831471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.009884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1988.121628] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 1989.716163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1989.747481] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 1990.556665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 1990.615614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp1 [ 1990.714801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp1 [ 2000.571755] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:16:05 (1786187765) [ 2003.521730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg417-server [ 2005.803181] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:16:11 (1786187771) [ 2008.371897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg417-server [ 2010.771037] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:16:16 (1786187776) [ 2013.287914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg417-server [ 2016.351423] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:16:21 (1786187781) [ 2019.438488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg417-server [ 2022.230405] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:16:27 (1786187787) [ 2024.666947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg417-server [ 2026.845089] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:16:32 (1786187792) [ 2029.311990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg417-server [ 2031.484623] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:16:36 (1786187796) [ 2033.829266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg417-server [ 2035.811772] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:16:41 (1786187801) [ 2037.677712] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg417-server [ 2039.786759] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:16:45 (1786187805) [ 2041.538724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg417-server [ 2043.649705] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:16:49 (1786187809) [ 2043.889839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.935490] LNet: 82616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.950843] LNetError: 82616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.961051] LNet: Removed LNI 192.168.204.17@tcp [ 2045.030824] LNet: Removed LNI 192.168.204.17@tcp1 [ 2045.087907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.172614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2045.198570] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2046.044322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2046.231720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2046.547466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.590331] LNet: 82763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.601724] LNetError: 82763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.615730] LNetError: 82763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2046.637086] LNet: Removed LNI 192.168.204.17@tcp [ 2046.687791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.783334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2046.810853] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2047.642853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2048.028079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2048.067622] LNetError: 82908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.125547] LNet: Removed LNI 192.168.204.17@tcp [ 2049.142807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2049.190353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2049.219263] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2050.611780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2050.793757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2051.207200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.253933] LNet: 83056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.266812] LNet: 83056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2051.278987] LNetError: 83056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.328642] LNet: Removed LNI 192.168.204.17@tcp [ 2052.388559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.496650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2052.534424] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2053.493585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2053.768422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.886996] LNet: Removed LNI 192.168.204.17@tcp [ 2054.936958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.015863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.049598] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2056.208685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2056.309507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2056.519388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2056.560750] LNet: 83349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.577186] LNet: 83349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2056.593767] LNetError: 83349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.610746] LNetError: 83349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2056.637824] LNet: Removed LNI 192.168.204.17@tcp [ 2056.700046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.796067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2056.843669] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2057.852954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2058.164704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.237451] LNet: Removed LNI 192.168.204.17@tcp [ 2059.286171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.413883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.442683] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2060.586200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2060.651663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2060.913271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.993542] LNet: Removed LNI 192.168.204.17@tcp [ 2062.035419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.118544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2062.171054] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2063.181432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2063.565890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2064.615402] LNet: Removed LNI 192.168.204.17@tcp [ 2064.662257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.753161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.784723] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2065.652249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2065.802306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2066.231330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.258792] LNet: 83934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.269396] LNet: 83934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2066.284238] LNetError: 83934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2066.294502] LNetError: 83934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2066.319917] LNet: Removed LNI 192.168.204.17@tcp [ 2066.376491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.501599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2066.543593] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2067.763736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2069.302152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2070.374765] LNet: Removed LNI 192.168.204.17@tcp [ 2070.401662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.455898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2070.482663] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2071.180961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2071.271814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2071.590874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.678595] LNet: Removed LNI 192.168.204.17@tcp [ 2072.718512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.809539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2072.847816] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2073.927833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2074.376039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2075.494442] LNet: Removed LNI 192.168.204.17@tcp [ 2075.518661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.584662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2075.615249] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2076.409659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2076.480860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.508105] LNet: Removed LNI 192.168.204.17@tcp [ 2076.523902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.596972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.631243] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2077.500360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2077.595329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.620218] LNet: Removed LNI 192.168.204.17@tcp [ 2077.656077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.725332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2077.811414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.877402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.948963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2078.014633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.046244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.089877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2078.167507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.200956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.247807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2078.268715] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2079.203758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.248920] LNet: Removed LNI 192.168.204.17@tcp [ 2079.274489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.391476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2088.398683] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:17:33 (1786187853) [ 2088.500800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.526575] LNet: 85395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.533246] LNet: 85395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 2088.604672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.724458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2088.810567] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/180] [ 2090.598468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.633985] LNetError: 85540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.640380] LNetError: 85540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2090.654256] LNet: Removed LNI 192.168.204.17@tcp231 [ 2090.686354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.779729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2090.809927] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/180] [ 2090.819392] LNet: 85548:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2091.277870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2091.304159] LNet: Removed LNI 192.168.204.17@tcp231 [ 2091.322935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2091.353090] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/180] [ 2092.662846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.690674] LNet: Removed LNI 192.168.204.17@tcp231 [ 2092.711395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.777855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2092.800726] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/0] [ 2092.807523] LNet: 85701:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2093.230842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2093.257864] LNet: Removed LNI 192.168.204.17@tcp231 [ 2093.278773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2093.315720] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/0] [ 2094.460311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.497126] LNet: Removed LNI 192.168.204.17@tcp231 [ 2094.516796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.575861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2094.597660] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/60] [ 2094.604606] LNet: 85852:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2094.901798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2094.927866] LNet: Removed LNI 192.168.204.17@tcp231 [ 2094.940465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2094.958302] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/60] [ 2095.917434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.974938] LNet: Removed LNI 192.168.204.17@tcp231 [ 2096.024135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.100500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2096.123790] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/180] [ 2096.582645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2096.608627] LNet: Removed LNI 192.168.204.17@tcp231 [ 2096.627919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2096.652915] LNet: Added LNI 192.168.204.17@tcp231 [8/256/0/180] [ 2105.099981] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:17:50 (1786187870) [ 2105.674215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.739501] LNet: Removed LNI 192.168.204.17@tcp231 [ 2105.782327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.877985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2105.904096] LNet: Added LNI 192.168.204.17@tcp232 [8/256/0/180] [ 2106.845798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2106.920906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2106.957972] LNet: Removed LNI 192.168.204.17@tcp232 [ 2106.982724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2107.012511] LNet: Added LNI 192.168.204.17@tcp232 [8/256/0/180] [ 2107.856806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2107.913297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2107.954742] LNet: Removed LNI 192.168.204.17@tcp232 [ 2107.977539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2108.013558] LNet: Added LNI 192.168.204.17@tcp232 [8/256/0/180] [ 2108.738112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2108.780656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2108.819707] LNet: Removed LNI 192.168.204.17@tcp232 [ 2108.837890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2108.871547] LNet: Added LNI 192.168.204.17@tcp232 [8/256/0/180] [ 2109.923608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2109.955489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2109.983860] LNet: Removed LNI 192.168.204.17@tcp232 [ 2110.005878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2118.217191] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:18:03 (1786187883) [ 2119.474516] Key type .llcrypt unregistered [ 2119.478939] Key type ._llcrypt unregistered [ 2127.162706] Key type ._llcrypt registered [ 2127.165731] Key type .llcrypt registered [ 2127.343647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2127.361640] alg: No test for adler32 (adler32-zlib) [ 2128.408697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2128.485381] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2137.283341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 2148.314587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.415738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2148.481934] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 2150.236647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_interrupt -r 2 [ 2150.289452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_interrupt -r 2 [ 2150.329959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_interrupt -r 2 [ 2150.423199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_interrupt -r 2 [ 2150.484196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_interrupt -r 2 [ 2150.527148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_interrupt -r 2 [ 2150.555976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_interrupt -r 2 [ 2150.609676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_interrupt -r 2 [ 2150.668858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2150.706556] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 2150.720843] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 2151.177635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2151.273652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2151.334521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2151.372921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2151.422361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2151.480670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2151.554482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2151.590667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2151.628440] LNet: 88512:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.653242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2151.683185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2151.714310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2151.750367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2151.787685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2151.825355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2152.406261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_dropped -r 2 [ 2152.442956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_dropped -r 2 [ 2152.477749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_dropped -r 2 [ 2152.521735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_dropped -r 2 [ 2152.591609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_dropped -r 2 [ 2152.715315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_dropped -r 2 [ 2152.817695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_dropped -r 2 [ 2152.880703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_dropped -r 2 [ 2152.956734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2153.002840] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 2153.591170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2153.672445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2153.749650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2153.818228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2153.872357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2153.923816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2153.978911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.058431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.106928] LNet: 88632:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.152674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.207332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.265676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.316553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.370678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.407583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2155.133068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_aborted -r 2 [ 2155.237180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_aborted -r 2 [ 2155.313105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_aborted -r 2 [ 2155.392038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_aborted -r 2 [ 2155.463255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_aborted -r 2 [ 2155.524934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_aborted -r 2 [ 2155.587690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_aborted -r 2 [ 2155.632840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_aborted -r 2 [ 2155.692460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2155.719608] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 2156.068596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.114678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2156.155318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2156.201812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2156.238580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2156.269797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2156.299919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2156.344633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2156.375282] LNet: 88753:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2156.407620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2156.450690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2156.508456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2156.557691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2156.598308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.639409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2157.482023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_no_route -r 2 [ 2157.511508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_no_route -r 2 [ 2157.544759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_no_route -r 2 [ 2157.583844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_no_route -r 2 [ 2157.621369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_no_route -r 2 [ 2157.685358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_no_route -r 2 [ 2157.742236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_no_route -r 2 [ 2157.789876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_no_route -r 2 [ 2157.823355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2157.845202] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 2158.284066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2158.368722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2158.455146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2158.528796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2158.604459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2158.681738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2158.762118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2158.793281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2158.809475] LNet: 88873:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.826518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2158.858794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2158.893319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2158.925280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2158.957340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2159.012347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2159.942981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e local_timeout -r 2 [ 2159.996729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e local_timeout -r 2 [ 2160.047238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e local_timeout -r 2 [ 2160.088722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e local_timeout -r 2 [ 2160.136587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e local_timeout -r 2 [ 2160.175113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e local_timeout -r 2 [ 2160.245750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e local_timeout -r 2 [ 2160.301970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e local_timeout -r 2 [ 2160.352748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2160.696301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2160.742132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2160.780884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2160.807963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2160.835420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2160.869285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2160.906167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2160.941464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2160.984295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2161.056760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2161.128534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2161.164482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2161.197532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2161.223659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.839733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp -m GET -e remote_dropped -r 2 [ 2161.883403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.117@tcp1 -m GET -e remote_dropped -r 2 [ 2161.926768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp -m GET -e remote_dropped -r 2 [ 2161.962330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp -d 192.168.204.17@tcp1 -m GET -e remote_dropped -r 2 [ 2162.013734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp -m GET -e remote_dropped -r 2 [ 2162.053281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.117@tcp1 -m GET -e remote_dropped -r 2 [ 2162.128768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp -m GET -e remote_dropped -r 2 [ 2162.165773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.17@tcp1 -d 192.168.204.17@tcp1 -m GET -e remote_dropped -r 2 [ 2162.197772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.17@tcp 192.168.204.117@tcp [ 2162.230688] LNet: There was an unexpected network error while writing to 192.168.204.117: rc = -22 [ 2162.238590] LNet: Skipped 1 previous similar message [ 2162.243139] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.117@tcp [ 2162.712710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2162.780756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2162.806520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2162.831847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2162.856970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2162.883431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2162.922488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2162.965631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2162.994204] LNet: 89113:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.998774] LNet: 89113:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2163.018686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2163.050461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2163.091929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2163.150440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2163.204644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2163.250491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2170.304248] LNet: 89192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.310272] LNetError: 89192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.323425] LNet: Removed LNI 192.168.204.17@tcp [ 2170.343767] LNet: Removed LNI 192.168.204.17@tcp1 [ 2170.762749] Key type .llcrypt unregistered [ 2170.773663] Key type ._llcrypt unregistered [ 2195.980155] Key type ._llcrypt registered [ 2195.986454] Key type .llcrypt registered [ 2197.717571] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:19:22 (1786187962) [ 2198.208090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.242907] alg: No test for adler32 (adler32-zlib) [ 2199.244586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.323691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.347608] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2200.257685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2200.282954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.17@tcp [ 2200.298376] LNetError: 90182:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.17@tcp to recovery [ 2200.390354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.17@tcp [ 2200.495463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2207.825750] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:19:33 (1786187973) [ 2207.957241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.995176] LNet: 90767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.014352] LNetError: 90767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.037112] LNet: Removed LNI 192.168.204.17@tcp [ 2208.066387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.146834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2208.170789] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2208.820927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2208.836314] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 2209.947243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.17@tcp [ 2209.983619] LNetError: 91039:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.204.17@tcp to recovery [ 2210.147600] LNet: 1 local NIs in recovery (showing 1): 192.168.204.17@tcp [ 2210.164482] LNet: There was an unexpected network error while writing to 192.168.204.17: rc = -22 [ 2211.456164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.17@tcp [ 2221.453751] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:19:46 (1786187986) [ 2223.655324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg417-server [ 2225.667945] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:19:51 (1786187991) [ 2227.650661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg417-server [ 2229.699712] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:19:55 (1786187995) [ 2231.833936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg417-server [ 2234.269941] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:19:59 (1786187999) [ 2234.445776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.525726] LNet: 92658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.540853] LNetError: 92658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.551162] LNet: Removed LNI 192.168.204.17@tcp [ 2235.558645] LNetError: 92658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.581979] LNet: Removed LNI 192.168.204.17@tcp1 [ 2235.620235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.741939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.820957] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2237.027594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2237.111948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2237.162422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2237.229959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2237.274594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2237.345530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2246.149138] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:20:10 (1786188010) [ 2246.418474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.519888] LNet: 93382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.546895] LNetError: 93382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.579372] LNet: Removed LNI 192.168.204.17@tcp [ 2246.638942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.733464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.758769] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2257.746672] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:20:22 (1786188022) [ 2257.909253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.970109] LNet: 94086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.976527] LNetError: 94086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.000780] LNet: Removed LNI 192.168.204.17@tcp [ 2258.052319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.158869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2258.184175] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2268.014511] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:20:32 (1786188032) [ 2269.928612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2271.767656] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:20:37 (1786188037) [ 2272.538103] LNet: 95129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.542763] LNetError: 95129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.567601] LNet: Removed LNI 192.168.204.17@tcp [ 2273.221940] Key type .llcrypt unregistered [ 2273.223639] Key type ._llcrypt unregistered [ 2281.340757] Key type ._llcrypt registered [ 2281.343318] Key type .llcrypt registered [ 2281.796024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2281.828495] alg: No test for adler32 (adler32-zlib) [ 2282.899341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2282.980548] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2292.740657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 2298.775334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.117@tcp [ 2299.388909] LNet: 95562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.400040] LNetError: 95562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.415961] LNet: Removed LNI 192.168.204.17@tcp [ 2300.242958] Key type .llcrypt unregistered [ 2300.247568] Key type ._llcrypt unregistered [ 2315.133299] Key type ._llcrypt registered [ 2315.138527] Key type .llcrypt registered [ 2316.685193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2318.221717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2320.326579] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:21:25 (1786188085) [ 2321.982942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2323.745025] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:21:29 (1786188089) [ 2325.859046] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg417-server [ 2327.726697] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:21:33 (1786188093) [ 2329.293287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg417-server [ 2331.057239] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:21:36 (1786188096) [ 2333.432736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2333.474410] alg: No test for adler32 (adler32-zlib) [ 2334.472720] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 2334.733124] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2336.408904] Key type lgssc registered [ 2338.431141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2362.817321] Key type lgssc unregistered [ 2363.274392] LNet: 98743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.283526] LNetError: 98743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.308199] LNet: Removed LNI 192.168.204.17@tcp [ 2364.049678] Key type .llcrypt unregistered [ 2364.055146] Key type ._llcrypt unregistered [ 2381.836390] Key type ._llcrypt registered [ 2381.840432] Key type .llcrypt registered [ 2383.701745] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:22:28 (1786188148) [ 2384.251385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2384.274850] alg: No test for adler32 (adler32-zlib) [ 2385.318862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.455771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2385.579655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2385.640257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2385.757228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2395.407549] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:22:40 (1786188160) [ 2395.838511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.925625] LNet: 100160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2395.973270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.056748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.082159] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2397.123569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.156537] LNet: 100305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.166320] LNetError: 100305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.184268] LNet: Removed LNI 192.168.204.17@tcp [ 2397.227474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.293349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2397.318854] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2397.938990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2397.973364] LNetError: 100333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.989543] LNet: Removed LNI 192.168.204.17@tcp [ 2398.015665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.092866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2398.120876] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2406.564450] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:22:52 (1786188172) [ 2406.745058] LNet: 100923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2406.748592] LNet: 100923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2406.752840] LNetError: 100923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2406.764817] LNet: Removed LNI 192.168.204.17@tcp [ 2407.477807] Key type .llcrypt unregistered [ 2407.480548] Key type ._llcrypt unregistered [ 2416.062843] Key type ._llcrypt registered [ 2416.068619] Key type .llcrypt registered [ 2416.517717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2416.560827] alg: No test for adler32 (adler32-zlib) [ 2417.606960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.733792] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2417.737623] LNet: *** cfs_fail_loc=e003, val=100*** [ 2418.655598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2418.696732] LNet: *** cfs_fail_loc=e003, val=100*** [ 2418.701324] LNet: Skipped 2 previous similar messages [ 2418.714306] Ping reply received with mapping: [ 2418.718448] found metadata reply NI[192.168.204.17@tcp] [ 2418.725877] 11 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2418.735494] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2418.750365] 5a 5a 5a 5a 5a 5a 5a 5a [ 2419.036879] LNet: 101326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.054654] LNetError: 101326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.078812] LNet: Removed LNI 192.168.204.17@tcp [ 2419.824684] Key type .llcrypt unregistered [ 2419.828744] Key type ._llcrypt unregistered [ 2439.768597] Key type ._llcrypt registered [ 2439.770741] Key type .llcrypt registered [ 2441.272930] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:23:26 (1786188206) [ 2441.898733] Key type .llcrypt unregistered [ 2441.900549] Key type ._llcrypt unregistered [ 2453.778511] Key type ._llcrypt registered [ 2453.782936] Key type .llcrypt registered [ 2454.270935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2454.310752] alg: No test for adler32 (adler32-zlib) [ 2455.521964] LNetError: 102386:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2456.122830] Key type .llcrypt unregistered [ 2456.131380] Key type ._llcrypt unregistered [ 2465.835738] Key type ._llcrypt registered [ 2465.838198] Key type .llcrypt registered [ 2467.721849] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:23:52 (1786188232) [ 2468.843305] Key type .llcrypt unregistered [ 2468.845369] Key type ._llcrypt unregistered [ 2477.872168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2477.875700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2480.821560] Key type ._llcrypt registered [ 2480.824320] Key type .llcrypt registered [ 2481.150642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2481.181579] alg: No test for adler32 (adler32-zlib) [ 2482.205407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.260338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2494.180882] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:24:19 (1786188259) [ 2494.578687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.598709] LNet: 104180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.603320] LNetError: 104180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2494.618178] LNet: Removed LNI 10.1.2.3@tcp [ 2494.635804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.271294] Key type .llcrypt unregistered [ 2495.273570] Key type ._llcrypt unregistered [ 2502.967823] Key type ._llcrypt registered [ 2502.970732] Key type .llcrypt registered [ 2503.179974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2503.198897] alg: No test for adler32 (adler32-zlib) [ 2504.125696] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2504.190796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2504.319159] LNet: 104472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.324748] LNetError: 104472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.335780] LNet: Removed LNI 192.168.204.17@tcp [ 2505.346849] Key type .llcrypt unregistered [ 2505.350893] Key type ._llcrypt unregistered [ 2522.962578] Key type ._llcrypt registered [ 2522.966596] Key type .llcrypt registered [ 2524.742667] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:24:49 (1786188289) [ 2525.643627] Key type .llcrypt unregistered [ 2525.648078] Key type ._llcrypt unregistered [ 2533.151577] Key type ._llcrypt registered [ 2533.157216] Key type .llcrypt registered [ 2533.603382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2533.627355] alg: No test for adler32 (adler32-zlib) [ 2542.475700] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:25:08 (1786188308) [ 2542.806683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2542.872409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2542.962756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2543.004734] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2543.773767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2543.810260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2543.846346] LNet: Removed LNI 192.168.204.17@tcp [ 2544.276685] LNet: 106231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.842466] Key type .llcrypt unregistered [ 2544.844099] Key type ._llcrypt unregistered [ 2560.429987] Key type ._llcrypt registered [ 2560.433872] Key type .llcrypt registered [ 2561.918721] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:25:27 (1786188327) [ 2563.078856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2563.119403] alg: No test for adler32 (adler32-zlib) [ 2564.199424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2564.309514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2564.353362] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2565.400606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2565.465248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.17@tcp [ 2565.496043] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2565.508616] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2565.515910] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2565, 0, 2615 [ 2565.523133] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 2565.528535] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2565.536629] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2565.544289] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2565, 0, 2615 [ 2565.556607] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 2572.591477] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:25:38 (1786188338) [ 2573.280609] LNet: 107772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.289530] LNetError: 107772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.313910] LNet: Removed LNI 192.168.204.17@tcp [ 2574.119706] Key type .llcrypt unregistered [ 2574.122594] Key type ._llcrypt unregistered [ 2580.287420] Key type ._llcrypt registered [ 2580.300248] Key type .llcrypt registered [ 2580.579825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2580.595850] alg: No test for adler32 (adler32-zlib) [ 2581.566450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2581.592562] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2590.463580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 2603.439389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.518932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2603.552152] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 2604.919075] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2605.112207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2605.124788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2605.135553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2607.702942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2607.732539] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2621.942239] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:26:27 (1786188387) [ 2622.132941] LNet: 109284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.139368] LNetError: 109284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.155949] LNet: Removed LNI 192.168.204.17@tcp [ 2623.199159] LNet: Removed LNI 192.168.204.17@tcp1 [ 2623.202751] LNetError: 109284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2623.217093] LNetError: 109284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2623.236847] LNet: Removed LNI 10.1.2.3@tcp99 [ 2623.954880] Key type .llcrypt unregistered [ 2623.958434] Key type ._llcrypt unregistered [ 2634.530401] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2634.558300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2637.521250] Key type ._llcrypt registered [ 2637.526167] Key type .llcrypt registered [ 2637.821438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2637.842846] alg: No test for adler32 (adler32-zlib) [ 2638.780418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2638.846349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2638.869780] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2639.934701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2639.959470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2642.168933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2642.223887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2642.286429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2642.305789] LNetError: 110038:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2642.354330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2642.405034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2642.811375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2642.938849] LNet: 110059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.950334] LNetError: 110059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2642.960810] LNet: Removed LNI 10.1.2.3@tcp [ 2644.005533] LNet: Removed LNI 192.168.204.17@tcp [ 2644.851573] Key type .llcrypt unregistered [ 2644.857873] Key type ._llcrypt unregistered [ 2659.952504] Key type ._llcrypt registered [ 2659.954976] Key type .llcrypt registered [ 2661.921569] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:27:06 (1786188426) [ 2662.605267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.618293] alg: No test for adler32 (adler32-zlib) [ 2663.685248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2663.739753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2663.759728] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2664.350812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2664.473608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg417-client@tcp [ 2672.901314] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:27:18 (1786188438) [ 2673.156101] LNet: 111593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2673.171228] LNetError: 111593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2673.213376] LNet: Removed LNI 192.168.204.17@tcp [ 2673.928823] Key type .llcrypt unregistered [ 2673.932142] Key type ._llcrypt unregistered [ 2683.241749] Key type ._llcrypt registered [ 2683.248386] Key type .llcrypt registered [ 2683.555978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2683.566183] alg: No test for adler32 (adler32-zlib) [ 2684.572633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2684.617778] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2684.670312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2693.238340] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:27:38 (1786188458) [ 2695.454629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2697.510782] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:27:42 (1786188462) [ 2699.849675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2701.698484] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:27:47 (1786188467) [ 2701.739494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2701.751676] LNet: 113016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2701.755411] LNetError: 113016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2701.783660] LNet: Removed LNI 192.168.204.17@tcp [ 2701.811626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2701.870428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2701.890122] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2702.632793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2702.674270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2711.846964] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:27:57 (1786188477) [ 2712.160679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2712.249544] LNet: 113732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2712.254719] LNetError: 113732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2712.277821] LNet: Removed LNI 192.168.204.17@tcp [ 2712.344672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2712.402451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2712.466522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2713.216982] Key type .llcrypt unregistered [ 2713.219341] Key type ._llcrypt unregistered [ 2730.271695] Key type ._llcrypt registered [ 2730.274253] Key type .llcrypt registered [ 2731.969541] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:28:17 (1786188497) [ 2732.454124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.469323] alg: No test for adler32 (adler32-zlib) [ 2733.411612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.509812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2733.567163] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2734.705482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2734.831659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2744.463680] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:28:29 (1786188509) [ 2744.640915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2744.710386] LNet: 115288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2744.728192] LNetError: 115288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2744.808658] LNet: Removed LNI 192.168.204.17@tcp [ 2744.894659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2745.032776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2745.133148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2755.331233] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:28:40 (1786188520) [ 2755.448533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2755.484219] LNet: 115867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.549667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2755.684171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2756.429820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2756.502391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2756.629954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2757.307195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2757.403624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2757.568505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2757.614497] LNet: 115909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.667810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2757.747922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2758.315669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2758.388326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2758.522802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2759.090865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2759.122823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2768.456704] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:28:53 (1786188533) [ 2768.719704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2768.822610] LNet: 116505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2768.910200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2768.998665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2769.022713] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2769.847525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2769.869646] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2770.858067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2770.932621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2771.229523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2771.286883] LNetError: 116789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2771.303357] LNet: Removed LNI 192.168.204.17@tcp [ 2771.318746] LNet: Removed LNI 192.168.204.17@tcp2 [ 2771.361457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2771.424682] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2771.435344] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2771.449265] LNet: 116795:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2771.838657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2772.171420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2773.215318] LNet: Removed LNI 192.168.204.17@tcp [ 2773.218697] LNetError: 116832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2773.230146] LNetError: 116832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2773.258480] LNet: Removed LNI 192.168.204.17@tcp2 [ 2773.314280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2773.455838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2773.491431] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2774.412950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2774.434731] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2775.186718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2775.222336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2775.289891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2775.668646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2775.690763] LNet: 117117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.695781] LNet: 117117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2775.701497] LNetError: 117117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.707779] LNet: Removed LNI 192.168.204.17@tcp [ 2776.741618] LNet: Removed LNI 192.168.204.17@tcp2 [ 2776.783658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2776.880145] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2776.888059] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2776.898642] LNet: 117123:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2777.445453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2777.668072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2777.688792] LNet: Removed LNI 192.168.204.17@tcp [ 2778.729714] LNet: Removed LNI 192.168.204.17@tcp2 [ 2778.774230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2778.908796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2778.966111] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2780.196575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2780.216058] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2781.250805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2781.314193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2781.712401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2781.735621] LNetError: 117440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.741735] LNetError: 117440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2781.746517] LNet: Removed LNI 192.168.204.17@tcp [ 2781.757087] LNet: Removed LNI 192.168.204.17@tcp2 [ 2781.790567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2781.872733] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2781.894782] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2781.905319] LNet: 117446:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2782.535501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2782.805914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2783.839164] LNet: Removed LNI 192.168.204.17@tcp [ 2783.855849] LNet: Removed LNI 192.168.204.17@tcp2 [ 2783.911660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2784.028705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2784.074948] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2785.177824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2785.204086] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2786.387166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2786.450208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2786.546655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2786.892282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2786.911235] LNet: 117767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.915402] LNet: 117767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2787.935801] LNet: Removed LNI 192.168.204.17@tcp [ 2788.968918] LNet: Removed LNI 192.168.204.17@tcp2 [ 2789.038639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2789.246776] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2789.252992] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2789.260576] LNet: 117774:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2789.689160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2790.138306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2790.198646] LNetError: 117811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2790.212590] LNetError: 117811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2790.230126] LNet: Removed LNI 192.168.204.17@tcp [ 2791.275618] LNet: Removed LNI 192.168.204.17@tcp2 [ 2791.328633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2791.469044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2791.535081] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2792.923532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2792.943578] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2794.252956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2794.297335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2794.442719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2795.058364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2796.127376] LNet: Removed LNI 192.168.204.17@tcp [ 2796.147429] LNet: Removed LNI 192.168.204.17@tcp2 [ 2796.205502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2796.336243] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2796.357679] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2796.365145] LNet: 118114:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2796.951601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2797.488443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2798.559817] LNet: Removed LNI 192.168.204.17@tcp [ 2798.572372] LNet: Removed LNI 192.168.204.17@tcp2 [ 2798.657346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2798.797845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2798.846222] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2799.731858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2799.775249] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2800.870869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2800.921517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2800.998953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2801.131946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2801.620220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2802.720297] LNet: Removed LNI 192.168.204.17@tcp [ 2803.749501] LNet: Removed LNI 192.168.204.17@tcp2 [ 2803.782139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2803.856584] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2803.865186] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 2804.278472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2814.121726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2816.709702] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:29:41 (1786188581) [ 2816.912527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2816.939969] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.945662] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2816.951636] LNetError: 119235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2816.958176] LNetError: 119235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2816.967945] LNet: Removed LNI 192.168.204.17@tcp [ 2818.029708] LNet: Removed LNI 192.168.204.17@tcp2 [ 2818.073761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2818.556748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2818.596973] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2819.871715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2819.903454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2821.383955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2821.405466] LNet: Removed LNI 10.1.2.3@tcp [ 2821.417898] LNet: Removed LNI 192.168.204.17@tcp [ 2821.441768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2821.535743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2821.569281] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2821.572861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2821.974901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2821.999736] LNet: Removed LNI 192.168.204.17@tcp [ 2822.010415] LNet: Removed LNI 10.1.2.3@tcp [ 2831.597485] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:29:57 (1786188597) [ 2831.666533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2831.741241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2831.826988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2831.865268] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2832.825283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2832.867731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2832.902213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2832.941828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2832.977974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.029965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.073294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.106222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.142548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.189479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.224157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.17@tcp [ 2833.380253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2840.470773] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:30:06 (1786188606) [ 2840.565213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2841.640503] LNet: Removed LNI 192.168.204.17@tcp [ 2841.669682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2841.844046] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2842.023183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2842.037432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2842.041774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2844.401281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2844.446888] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2845.281951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2845.306169] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2849.181455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 2849.265951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2849.345295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.17@tcp -m PUT -e local_timeout -r 1 [ 2849.473799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2849.565865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2849.578131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2850.107953] LNetError: 120915:0:(lib-move.c:4801:lnet_parse()) 192.168.204.17@tcp, src 192.168.204.17@tcp: Dropping PUT (error -108 looking up sender) [ 2851.167714] LNet: 121379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2851.173235] LNet: 121379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2851.184144] LNetError: 121379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2851.201932] LNetError: 121379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2851.216243] LNet: Removed LNI 10.1.2.3@tcp [ 2852.264993] LNet: Removed LNI 192.168.204.17@tcp [ 2853.156926] Key type .llcrypt unregistered [ 2853.160556] Key type ._llcrypt unregistered [ 2872.554767] Key type ._llcrypt registered [ 2872.560968] Key type .llcrypt registered [ 2874.315774] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:30:39 (1786188639) [ 2874.959326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2874.971450] alg: No test for adler32 (adler32-zlib) [ 2876.025157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2876.633354] LNet: 122234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2877.749931] Key type .llcrypt unregistered [ 2877.752664] Key type ._llcrypt unregistered [ 2885.460757] Key type ._llcrypt registered [ 2885.464567] Key type .llcrypt registered [ 2885.766291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2885.804332] alg: No test for adler32 (adler32-zlib) [ 2886.820825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2886.877831] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2896.684090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 2897.130834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2898.152864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.117@tcp [ 2898.208539] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.117@tcp [ 2898.629086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2898.705897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2898.752814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2898.799922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2898.851948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2898.882917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2898.916383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2898.971094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2899.011930] LNet: 122688:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2899.036201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2899.115492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2899.162415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2899.216075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2899.303040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2899.369384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2900.540877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2901.019538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2901.049872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2901.079113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2901.106954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2901.141111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2901.187373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2901.222737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2901.273168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2901.301253] LNet: 122806:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2901.324106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2901.366401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2901.406899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2901.459511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2901.496860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2901.552377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2902.233794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.117@tcp [ 2902.539442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2902.594975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2902.645077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2902.707582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2902.759334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2902.804367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2902.851592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2902.891959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2902.917856] LNet: 122923:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2902.941868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2902.977538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2903.011354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2903.055943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2903.110889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2903.138636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2904.020844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2904.450570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2904.499126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2904.535803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2904.573304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2904.617088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2904.687282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2904.742768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2904.828046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2904.943284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2905.027278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2905.108907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2905.140673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2905.177635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2905.235947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2905.580416] LNet: 123090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2905.584888] LNetError: 123090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2905.610541] LNet: Removed LNI 192.168.204.17@tcp [ 2905.908866] Key type .llcrypt unregistered [ 2905.911385] Key type ._llcrypt unregistered [ 2928.894422] Key type ._llcrypt registered [ 2928.896541] Key type .llcrypt registered [ 2930.647650] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:31:35 (1786188695) [ 2930.936677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.951870] alg: No test for adler32 (adler32-zlib) [ 2931.833267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2932.436952] LNet: 123978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.484373] Key type .llcrypt unregistered [ 2933.486082] Key type ._llcrypt unregistered [ 2942.416774] Key type ._llcrypt registered [ 2942.422229] Key type .llcrypt registered [ 2942.756257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2942.780966] alg: No test for adler32 (adler32-zlib) [ 2943.817498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2943.862151] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 2952.659459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.117@tcp [ 2963.283185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2963.364717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2963.403401] LNet: Added LNI 192.168.204.17@tcp1 [8/256/0/180] [ 2964.411914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2965.642716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.117@tcp [ 2965.940891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2965.976793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2966.008958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2966.041535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2966.068726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2966.094113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2966.125331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2966.154901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2966.173398] LNet: 124635:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2966.192241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2966.225449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2966.265326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2966.338022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2966.426894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2966.436279] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.117@tcp [ 2966.483690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2967.289677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2967.803022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2967.847248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2967.891041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2967.942560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2968.006581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2968.076766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2968.152197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2968.220518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2968.273490] LNet: 124752:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2968.324444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2968.382266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2968.465953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2968.555848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2968.595720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2968.642868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2970.198182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.117@tcp [ 2970.691660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2970.761639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2970.816621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2970.870304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2970.920732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2970.972839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2971.041148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2971.085297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2971.127704] LNet: 124870:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2971.181712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2971.224805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2971.266876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2971.319519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2971.369449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2971.405646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2972.355839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2972.769903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2972.835602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2972.890918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2972.924590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2972.954748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2972.982305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2973.026462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2973.075193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2973.131913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2973.186967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2973.236941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2973.314300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2973.387986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2973.432290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2979.162355] LNet: 125072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2979.175742] LNetError: 125072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2979.183769] LNet: Removed LNI 192.168.204.17@tcp [ 2979.199317] LNet: Removed LNI 192.168.204.17@tcp1 [ 2979.562914] Key type .llcrypt unregistered [ 2979.566632] Key type ._llcrypt unregistered [ 2998.970688] Key type ._llcrypt registered [ 2998.974771] Key type .llcrypt registered [ 3000.782260] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:32:45 (1786188765) [ 3001.332202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.347457] alg: No test for adler32 (adler32-zlib) [ 3002.263198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3002.311904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3002.330947] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 3003.211509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3003.237861] LNet: Added LNI 192.168.204.17@tcp2 [8/256/0/180] [ 3004.154986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.17@tcp [ 3004.228205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3005.279338] LNet: Removed LNI 192.168.204.17@tcp [ 3012.007950] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:32:57 (1786188777) [ 3013.383143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3015.261410] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:33:00 (1786188780) [ 3022.629335] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:33:08 (1786188788) [ 3030.308324] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:33:16 (1786188796) [ 3039.959544] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:33:24 (1786188804) [ 3048.818371] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:33:33 (1786188813) [ 3056.081420] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:33:42 (1786188822) [ 3064.416661] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:33:49 (1786188829) [ 3071.912831] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:33:57 (1786188837) [ 3079.594665] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:34:05 (1786188845) [ 3089.781566] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:34:14 (1786188854) [ 3100.867044] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:34:26 (1786188866) [ 3110.108830] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:34:35 (1786188875) [ 3118.506055] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:34:43 (1786188883) [ 3126.628269] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:34:52 (1786188892) [ 3134.207167] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:35:00 (1786188900) [ 3141.092429] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:35:06 (1786188906) [ 3150.405145] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:35:15 (1786188915) [ 3159.212816] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:35:24 (1786188924) [ 3159.524356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3159.627943] LNet: 136770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3159.641815] LNetError: 136770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3159.681400] LNet: Removed LNI 192.168.204.17@tcp2 [ 3159.750080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3160.037268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3160.066236] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 3161.558715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3161.605130] LNet: 136913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3161.614992] LNetError: 136913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3161.626650] LNet: Removed LNI 192.168.204.17@tcp [ 3161.653736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3161.737308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3161.768563] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 3169.523806] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:35:35 (1786188935) [ 3178.033200] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:35:43 (1786188943) [ 3186.349852] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:35:52 (1786188952) [ 3194.828954] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:35:59 (1786188959) [ 3202.767441] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:36:08 (1786188968) [ 3211.898602] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:36:17 (1786188977) [ 3222.351657] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:36:27 (1786188987) [ 3230.836989] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:36:36 (1786188996) [ 3239.559527] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:36:44 (1786189004) [ 3247.962744] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:36:53 (1786189013) [ 3256.622512] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:37:01 (1786189021) [ 3264.535222] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:37:10 (1786189030) [ 3272.769649] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2974 sec ========== 07:37:18 (1786189038) [ 3273.137545] LNet: 144379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3273.146707] LNetError: 144379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3273.168507] LNet: Removed LNI 192.168.204.17@tcp [ 3274.005932] Key type .llcrypt unregistered [ 3274.009562] Key type ._llcrypt unregistered [ 3287.864606] Key type ._llcrypt registered [ 3287.870390] Key type .llcrypt registered [ 3288.309739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3288.322620] alg: No test for adler32 (adler32-zlib) [ 3289.488536] Lustre: Lustre: Build Version: 2.17.56_50_g336d872 [ 3289.828505] LNet: Added LNI 192.168.204.17@tcp [8/256/0/180] [ 3291.527176] Key type lgssc registered [ 3293.487577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3371.501706] Lustre: Mounted lustre-client - version 2.17.56_50_g336d872 [ 3377.793676] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3386.264514] Lustre: Unmounted lustre-client [ 3411.552707] Key type lgssc unregistered [ 3411.862370] LNet: 147343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.872523] LNetError: 147343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.900505] LNet: Removed LNI 192.168.204.17@tcp [ 3412.875272] Key type .llcrypt unregistered [ 3412.884608] Key type ._llcrypt unregistered