[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 446636045 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003340] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009004] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011122] LSM: Security Framework initializing [ 0.013014] Yama: becoming mindful. [ 0.013910] SELinux: Initializing. [ 0.015010] *** VALIDATE selinux *** [ 0.022486] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026601] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030109] *** VALIDATE tmpfs *** [ 0.032344] *** VALIDATE proc *** [ 0.033237] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043339] debug: unmapping init [mem 0xffffffffa1059000-0xffffffffa1060fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046708] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054313] rcu: Hierarchical SRCU implementation. [ 0.056443] smp: Bringing up secondary CPUs ... [ 0.057574] x86: Booting SMP configuration: [ 0.058030] .... node #0, CPUs: #1 #2 #3 [ 0.061318] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136020] node 0 deferred pages initialised in 69ms [ 0.139272] devtmpfs: initialized [ 0.140232] x86/mm: Memory block size: 128MB [ 0.142744] gcov: version magic: 0x41383552 [ 0.145276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147309] pinctrl core: initialized pinctrl subsystem [ 0.148198] [ 0.148866] ************************************************************* [ 0.149014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150014] ** ** [ 0.151018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152014] ** ** [ 0.153028] ** This means that this kernel is built to expose internal ** [ 0.154016] ** IOMMU data structures, which may compromise security on ** [ 0.155014] ** your system. ** [ 0.156013] ** ** [ 0.157017] ** If you see this message and you are not debugging the ** [ 0.158016] ** kernel, report this immediately to your vendor! ** [ 0.159017] ** ** [ 0.160015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161014] ************************************************************* [ 0.162616] NET: Registered protocol family 16 [ 0.163451] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.164089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.166459] cpuidle: using governor menu [ 0.168933] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172758] PCI: Using configuration type 1 for base access [ 0.175219] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184190] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.186135] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190069] cryptd: max_cpu_qlen set to 1000 [ 0.194087] ACPI: Added _OSI(Module Device) [ 0.195028] ACPI: Added _OSI(Processor Device) [ 0.196024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198042] ACPI: Added _OSI(Processor Aggregator Device) [ 0.203784] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.213148] ACPI: Interpreter enabled [ 0.214080] ACPI: PM: (supports S0 S3 S4 S5) [ 0.215021] ACPI: Using IOAPIC for interrupt routing [ 0.217256] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.222395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232261] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.234044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246460] acpiphp: Slot [2] registered [ 0.247134] acpiphp: Slot [5] registered [ 0.249204] acpiphp: Slot [6] registered [ 0.250120] acpiphp: Slot [3] registered [ 0.252109] acpiphp: Slot [4] registered [ 0.253109] acpiphp: Slot [7] registered [ 0.255158] acpiphp: Slot [8] registered [ 0.256128] acpiphp: Slot [9] registered [ 0.257110] acpiphp: Slot [10] registered [ 0.259112] acpiphp: Slot [11] registered [ 0.260113] acpiphp: Slot [12] registered [ 0.261104] acpiphp: Slot [13] registered [ 0.262140] acpiphp: Slot [14] registered [ 0.263117] acpiphp: Slot [15] registered [ 0.265111] acpiphp: Slot [16] registered [ 0.266129] acpiphp: Slot [17] registered [ 0.268167] acpiphp: Slot [18] registered [ 0.269135] acpiphp: Slot [19] registered [ 0.271180] acpiphp: Slot [20] registered [ 0.272112] acpiphp: Slot [21] registered [ 0.273134] acpiphp: Slot [22] registered [ 0.275133] acpiphp: Slot [23] registered [ 0.276137] acpiphp: Slot [24] registered [ 0.278133] acpiphp: Slot [25] registered [ 0.280116] acpiphp: Slot [26] registered [ 0.281167] acpiphp: Slot [27] registered [ 0.283144] acpiphp: Slot [28] registered [ 0.284154] acpiphp: Slot [29] registered [ 0.285219] acpiphp: Slot [30] registered [ 0.287133] acpiphp: Slot [31] registered [ 0.288080] PCI host bridge to bus 0000:00 [ 0.290032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292047] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309115] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314647] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323549] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.328459] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.337490] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.352910] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.362681] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373988] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429474] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.442017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.471547] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.474485] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.477429] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.480345] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.485190] iommu: Default domain type: Passthrough [ 0.487652] SCSI subsystem initialized [ 0.489202] ACPI: bus type USB registered [ 0.490147] usbcore: registered new interface driver usbfs [ 0.493146] usbcore: registered new interface driver hub [ 0.495123] usbcore: registered new device driver usb [ 0.498251] pps_core: LinuxPPS API ver. 1 registered [ 0.500014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.503091] PTP clock support registered [ 0.505185] EDAC MC: Ver: 3.0.0 [ 0.507416] PCI: Using ACPI for IRQ routing [ 0.509849] NetLabel: Initializing [ 0.511014] NetLabel: domain hash size = 128 [ 0.512010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.514083] NetLabel: unlabeled traffic allowed by default [ 0.516169] vgaarb: loaded [ 0.517354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.519016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.526399] clocksource: Switched to clocksource kvm-clock [ 0.636100] VFS: Disk quotas dquot_6.6.0 [ 0.637863] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.640659] *** VALIDATE ramfs *** [ 0.641991] *** VALIDATE hugetlbfs *** [ 0.643697] pnp: PnP ACPI init [ 0.646225] pnp: PnP ACPI: found 6 devices [ 0.665321] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.669493] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.671604] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.673600] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.675611] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.678020] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.680973] NET: Registered protocol family 2 [ 0.683453] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.688195] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.690913] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.696379] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.700484] TCP: Hash tables configured (established 65536 bind 65536) [ 0.703521] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.706612] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.709455] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.712713] NET: Registered protocol family 1 [ 0.715660] RPC: Registered named UNIX socket transport module. [ 0.718032] RPC: Registered udp transport module. [ 0.720599] RPC: Registered tcp transport module. [ 0.722728] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.725743] NET: Registered protocol family 44 [ 0.727477] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.729958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.732257] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.734926] PCI: CLS 0 bytes, default 64 [ 0.736807] Unpacking initramfs... [ 2.210380] debug: unmapping init [mem 0xffff93a8fcc64000-0xffff93a8fffcffff] [ 2.217588] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.220386] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225051] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.764438] Initialise system trusted keyrings [ 2.766526] Key type blacklist registered [ 2.768707] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.777895] zbud: loaded [ 2.781050] *** VALIDATE nfs *** [ 2.782449] *** VALIDATE nfs4 *** [ 2.784172] pstore: using deflate compression [ 2.787887] Platform Keyring initialized [ 2.910214] NET: Registered protocol family 38 [ 2.912406] Key type asymmetric registered [ 2.914118] Asymmetric key parser 'x509' registered [ 2.916234] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.919119] io scheduler mq-deadline registered [ 2.920671] io scheduler kyber registered [ 2.922165] io scheduler bfq registered [ 2.924020] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.926936] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.929835] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.932763] ACPI: Power Button [PWRF] [ 2.938168] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.945224] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.965788] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.993386] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.020453] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.026676] Non-volatile memory driver v1.3 [ 3.028273] Linux agpgart interface v0.103 [ 3.063662] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.065875] vda: detected capacity change from 0 to 75083776 [ 3.084700] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.088018] vdb: detected capacity change from 0 to 1073741824 [ 3.097969] libphy: Fixed MDIO Bus: probed [ 3.105289] usbcore: registered new interface driver usbserial_generic [ 3.107980] usbserial: USB Serial support registered for generic [ 3.110398] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.114984] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.117317] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.120050] mousedev: PS/2 mouse device common for all mice [ 3.122787] rtc_cmos 00:05: RTC can wake from S4 [ 3.126728] rtc_cmos 00:05: registered as rtc0 [ 3.127643] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.128618] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.135532] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.135961] intel_pstate: CPU model not supported [ 3.140216] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.144962] hid: raw HID events driver (C) Jiri Kosina [ 3.147204] usbcore: registered new interface driver usbhid [ 3.149285] usbhid: USB HID core driver [ 3.151034] drop_monitor: Initializing network drop monitor service [ 3.153676] Initializing XFRM netlink socket [ 3.155882] NET: Registered protocol family 10 [ 3.159435] Segment Routing with IPv6 [ 3.160932] NET: Registered protocol family 17 [ 3.162927] mpls_gso: MPLS GSO support [ 3.169805] RAS: Correctable Errors collector initialized. [ 3.174899] AVX version of gcm_enc/dec engaged. [ 3.176545] AES CTR mode by8 optimization enabled [ 3.263497] sched_clock: Marking stable (3263459549, 0)->(4179411397, -915951848) [ 3.267442] registered taskstats version 1 [ 3.269524] Loading compiled-in X.509 certificates [ 3.271560] zswap: loaded using pool lzo/zbud [ 3.298418] Key type big_key registered [ 3.311930] Key type encrypted registered [ 3.313249] ima: No TPM chip found, activating TPM-bypass! [ 3.315437] ima: Allocated hash algorithm: sha1 [ 3.316977] ima: No architecture policies found [ 3.318934] evm: Initialising EVM extended attributes: [ 3.320644] evm: security.selinux [ 3.321714] evm: security.ima [ 3.322716] evm: security.capability [ 3.324099] evm: HMAC attrs: 0x1 [ 3.326746] rtc_cmos 00:05: setting system clock to 2026-09-07 10:06:54 UTC (1788775614) [ 3.333592] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 3.336717] debug: unmapping init [mem 0xffffffffa0d82000-0xffffffffa1058fff] [ 3.343152] Write protecting the kernel read-only data: 28672k [ 3.346680] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.349462] debug: unmapping init [mem 0xffffffff9fd14000-0xffffffff9fdfffff] [ 3.384835] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.396851] systemd[1]: Detected virtualization kvm. [ 3.399101] systemd[1]: Detected architecture x86-64. [ 3.401018] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.428867] systemd[1]: No hostname configured. [ 3.430183] systemd[1]: Set hostname to . [ 3.431794] random: systemd: uninitialized urandom read (16 bytes read) [ 3.434204] systemd[1]: Initializing machine ID from random generator. [ 3.497270] random: ln: uninitialized urandom read (6 bytes read) [ 3.600373] random: systemd: uninitialized urandom read (16 bytes read) [ 3.603300] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.608904] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.620963] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.290407] device-mapper: uevent: version 1.0.3 [ 4.292845] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.075962] virtio_net virtio0 ens2: renamed from eth0 [ 5.185168] scsi host0: ata_piix [ 5.242645] scsi host1: ata_piix [ 5.247597] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.250264] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.899376] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.829745] random: crng init done [ 9.833934] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.321335] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.544451] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.841846] SELinux: Disabled at runtime. [ 11.911979] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.921418] systemd[1]: Detected virtualization kvm. [ 11.923724] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.456641] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.461515] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.466520] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.471340] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.475329] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.482402] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.486761] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ 12.629273] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.924198] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.187131] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.245316] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.301173] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.318185] EDAC sbridge: Ver: 1.1.2 [ 14.324548] Key type dns_resolver registered [ 14.630497] NFS: Registering the id_resolver key type [ 14.633055] Key type id_resolver registered [ 14.634802] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg439-client login: [ 45.504371] libcfs: loading out-of-tree module taints kernel. [ 45.766117] Key type ._llcrypt registered [ 45.811040] Key type .llcrypt registered [ 46.457262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 46.466719] alg: No test for adler32 (adler32-zlib) [ 47.950251] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 48.940889] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 50.736188] Key type lgssc registered [ 52.690635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 183.446382] Lustre: Mounted lustre-client - version 2.17.57_103_g80273de [ 189.004464] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 201.288638] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing check_logdir /tmp/testlogs/ [ 208.012564] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing yml_node [ 209.379205] Lustre: lustre-OST0000-osc-ffff93a9495f1800: disconnect after 23s idle [ 209.835201] hrtimer: interrupt took 19856391 ns [ 213.619110] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 217.786994] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 220.943304] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 222.817956] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 06:10:32 EDT 2026 [ 239.959034] Lustre: Unmounted lustre-client [ 262.826840] Key type lgssc unregistered [ 263.176863] LNet: 8407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.184292] LNetError: 8407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.197348] LNet: Removed LNI 192.168.204.39@tcp [ 264.296148] Key type .llcrypt unregistered [ 264.299423] Key type ._llcrypt unregistered [ 293.906036] Key type ._llcrypt registered [ 293.909921] Key type .llcrypt registered [ 294.521410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 294.547555] alg: No test for adler32 (adler32-zlib) [ 295.815985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 296.003883] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 296.062582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 302.476602] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 302.632644] LNet: 9279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.647218] LNetError: 9279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.692855] LNet: Removed LNI 192.168.204.39@tcp [ 303.629583] Key type .llcrypt unregistered [ 303.638225] Key type ._llcrypt unregistered [ 311.821224] Key type ._llcrypt registered [ 311.825377] Key type .llcrypt registered [ 313.205659] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:12:02 (1788775922) [ 313.445921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 313.458395] alg: No test for adler32 (adler32-zlib) [ 314.434809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.582192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.602064] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.623912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.684640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 314.701633] LNet: 9703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 322.269445] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:12:11 (1788775931) [ 322.394147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.425503] LNet: 10276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.449846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.561379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 330.456735] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:12:20 (1788775940) [ 330.749937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.792107] LNet: 10867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.825501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.900805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 338.231984] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:12:28 (1788775948) [ 338.603868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.640558] LNet: 11454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.662271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.755090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 345.158816] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:12:34 (1788775954) [ 345.244181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.303203] LNet: 12029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.346966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.507984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 346.082116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 353.292964] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:12:42 (1788775962) [ 353.388743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.418896] LNet: 12622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.453426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.576749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 354.111244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 354.439396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 362.428854] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:12:51 (1788775971) [ 362.481271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.543855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.760849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 372.072363] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:13:01 (1788775981) [ 372.365466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.418043] LNet: 13812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.430068] LNet: 13812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 372.472714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.592792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 372.675648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.198638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 373.284805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 374.061166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 374.117901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 374.572681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 374.617938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.232247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 375.300262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.809766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 375.852897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.505842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 376.571150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 377.437498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 377.485438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 378.158680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 378.266190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.001635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 379.044478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.383179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 379.425218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.112680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 380.163935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.966871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.079882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 390.451488] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:13:19 (1788775999) [ 390.584537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.651479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.761031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 390.794984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 398.057969] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:13:27 (1788776007) [ 398.172884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.258865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.410888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 398.452969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 406.362966] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:13:35 (1788776015) [ 406.433583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.458337] LNet: 15661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.475972] LNet: 15661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 406.501534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.618359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 406.652779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 413.789658] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:13:43 (1788776023) [ 413.943821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.032359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.126824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.183942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 421.031507] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:13:50 (1788776030) [ 421.254809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.286867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.364933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 421.422130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.952465] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:13:58 (1788776038) [ 429.141910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.206565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.337313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 429.383668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 437.637229] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:14:06 (1788776046) [ 437.782138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.838740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.001479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.854942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.272204] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:14:16 (1788776056) [ 447.546636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.589995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.823826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.213948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.676743] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:14:25 (1788776065) [ 456.836435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.882425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.047806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.377947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.574479] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:14:35 (1788776075) [ 465.774251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.853928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.465704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.754143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.488532] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:14:44 (1788776084) [ 474.598317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.641369] LNet: 20494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.650885] LNet: 20494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 474.695413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.758651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 474.821127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.592796] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:14:52 (1788776092) [ 482.737714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.794316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.918288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.997343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.576543] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:15:00 (1788776100) [ 490.928292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.978611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.081240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 498.445322] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:15:08 (1788776108) [ 498.671703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.711829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.802733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 504.961822] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:15:14 (1788776114) [ 505.050777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.119404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.224471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 505.786699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 512.571563] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:15:22 (1788776122) [ 512.806926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.839499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.928963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 512.979343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 520.618887] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:15:30 (1788776130) [ 520.732951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.797351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.909883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 520.994794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 528.511690] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:15:38 (1788776138) [ 528.698831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.775335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.836955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 528.867428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 535.918311] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:15:45 (1788776145) [ 536.101943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.186754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.282720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 536.333877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 542.747963] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:15:52 (1788776152) [ 542.932192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.986504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.037367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 543.091502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 543.104631] LNetError: 25814:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 543.141978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 543.167377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 549.026650] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:15:58 (1788776158) [ 549.277817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.311919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.404638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YgwlrKxGmI [ 549.465268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YgwlrKxGmI [ 549.511252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YgwlrKxGmI [ 555.057475] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:16:05 (1788776165) [ 555.114819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.147302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.214279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 555.251640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 555.283610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 561.054474] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 06:16:10 (1788776170) [ 561.263415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.323879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.406755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 561.430337] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 563.063471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 564.243910] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 566.072721] Key type lgssc registered [ 567.173725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.595511] LNetError: 27576:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.39@tcp99 has no available nets [ 573.544386] Key type lgssc unregistered [ 573.896859] LNetError: 28678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.913678] LNet: Removed LNI 192.168.204.39@tcp [ 574.582363] Key type .llcrypt unregistered [ 574.584731] Key type ._llcrypt unregistered [ 581.445603] Key type ._llcrypt registered [ 581.447880] Key type .llcrypt registered [ 582.639208] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:16:32 (1788776192) [ 583.217595] Key type .llcrypt unregistered [ 583.221694] Key type ._llcrypt unregistered [ 589.092955] Key type ._llcrypt registered [ 589.096566] Key type .llcrypt registered [ 589.314863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 589.328728] alg: No test for adler32 (adler32-zlib) [ 590.262214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.322402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 590.340180] LNet: 29589:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 590.346709] LNet: 29589:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 590.353288] LNet: 29589:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 590.366170] LNet: Message forwarding will be enabled in 2s [ 593.440639] LNet: Message forwarding enabled [ 593.857709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 593.898169] LNet: Message forwarding will stop in 2s to 6s [ 597.542463] LNet: Message forwarding disabled [ 598.513509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.535539] LNet: 29687:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 598.541838] LNet: 29687:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 598.547873] LNet: 29687:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 598.562551] LNet: Message forwarding will be enabled in 2s [ 598.773471] LNet: 29700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.589782] Key type .llcrypt unregistered [ 599.592157] Key type ._llcrypt unregistered [ 611.573934] Key type ._llcrypt registered [ 611.580130] Key type .llcrypt registered [ 612.785531] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:17:02 (1788776222) [ 613.229442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 613.244767] alg: No test for adler32 (adler32-zlib) [ 614.145543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.234414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 621.840602] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:17:11 (1788776231) [ 621.991425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.063366] LNet: 31095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 622.114581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.226192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 622.275348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 622.307255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 622.343946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 622.402954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 622.452863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 622.508624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 622.569922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 622.623971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 622.682371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 622.750891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 622.806668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 622.860855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 622.905178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 622.945371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 623.008573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 623.060521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 623.140506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 623.210993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 623.247488] LNetError: 31144:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 630.052779] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:17:19 (1788776239) [ 630.287858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.310475] LNet: 31715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.331364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.394622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 637.963623] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:17:27 (1788776247) [ 639.544681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg439-server [ 641.658352] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:17:31 (1788776251) [ 641.870755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.891514] LNet: 32646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.911426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.002767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.033497] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 642.925366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[40-44]@tcp [ 642.952830] LNet: 32779:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 643.003064] LNetError: 32650:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.42@tcp is being used as a gateway but routing feature is not turned on [ 643.020857] LNetError: 32650:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.42@tcp has gone from up to down [ 650.641865] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:17:40 (1788776260) [ 653.161676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg439-server [ 654.962990] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:17:44 (1788776264) [ 655.170576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.211871] LNet: 33715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.222901] LNetError: 33715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.246154] LNet: Removed LNI 192.168.204.39@tcp [ 655.285628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.368569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 655.408256] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 656.155510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[40-45/2]@tcp [ 656.175334] LNet: 33834:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 656.183207] LNet: 33834:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 656.205739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[40-45/2]@tcp [ 656.228493] LNetError: 33721:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.42@tcp is being used as a gateway but routing feature is not turned on [ 656.240943] LNetError: 33721:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 656.253559] LNetError: 33721:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.42@tcp has gone from up to down [ 656.265871] LNetError: 33721:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 663.789491] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:17:53 (1788776273) [ 663.911747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.957810] LNet: 34423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.967405] LNetError: 34423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.993813] LNet: Removed LNI 192.168.204.39@tcp [ 664.020569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.192863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 664.260798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.311337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.450917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 664.545077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.608306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.692303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 664.794738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.869710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.982779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.100888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.173859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.263065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 665.408156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.450935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.543622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 665.661274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.707275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.778684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 665.895408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.936683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.127217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.237368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.269450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.325558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 666.418798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.465301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.580631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 666.686402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.722824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.769607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 666.820730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.856024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.921716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 674.675401] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:18:04 (1788776284) [ 677.047663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg439-server [ 679.562889] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:18:08 (1788776288) [ 682.479190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg439-server [ 685.217567] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:18:14 (1788776294) [ 685.338319] LNet: 35820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.347921] LNet: 35820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 686.064119] Key type .llcrypt unregistered [ 686.066308] Key type ._llcrypt unregistered [ 693.339965] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.654225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 693.661164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 693.676554] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.413633] Key type ._llcrypt registered [ 696.416635] Key type .llcrypt registered [ 696.583184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.601097] alg: No test for adler32 (adler32-zlib) [ 697.583816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.667161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.726738] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 698.631680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.688345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.651569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 700.680288] LNet: Removed LNI 10.1.2.3@tcp [ 700.834766] LNet: 36547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.848773] LNetError: 36547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.892840] LNet: Removed LNI 192.168.204.39@tcp [ 701.843462] Key type .llcrypt unregistered [ 701.848740] Key type ._llcrypt unregistered [ 714.497657] Key type ._llcrypt registered [ 714.500355] Key type .llcrypt registered [ 716.533471] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:18:45 (1788776325) [ 717.276556] Key type .llcrypt unregistered [ 717.278622] Key type ._llcrypt unregistered [ 723.112201] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.262412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.273250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.286042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.194692] Key type ._llcrypt registered [ 725.199461] Key type .llcrypt registered [ 725.424859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 725.454889] alg: No test for adler32 (adler32-zlib) [ 726.479809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.573537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.595779] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 727.703580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.758520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.685576] LNet: 38031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.705528] LNetError: 38031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 731.711513] LNet: Removed LNI 10.1.2.3@tcp [ 731.734618] LNet: Removed LNI 192.168.204.39@tcp [ 732.399609] Key type .llcrypt unregistered [ 732.404802] Key type ._llcrypt unregistered [ 746.863852] Key type ._llcrypt registered [ 746.867669] Key type .llcrypt registered [ 748.224899] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:19:17 (1788776357) [ 748.743401] Key type .llcrypt unregistered [ 748.749261] Key type ._llcrypt unregistered [ 754.524954] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.757250] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.765234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.777455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.326896] Key type ._llcrypt registered [ 757.335480] Key type .llcrypt registered [ 757.649844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.668236] alg: No test for adler32 (adler32-zlib) [ 758.618400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.683542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 758.716933] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 760.584371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.603284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.465619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 762.492252] LNet: Removed LNI 10.1.2.3@tcp [ 762.509236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 762.558611] LNet: Removed LNI 10.1.2.31@tcp [ 762.583466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.626583] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.346907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 763.374229] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 764.212086] LNet: 39947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.217617] LNetError: 39947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 764.222172] LNet: Removed LNI 10.1.2.31@tcp [ 764.232534] LNet: Removed LNI 10.1.2.3@tcp [ 764.937831] Key type .llcrypt unregistered [ 764.939588] Key type ._llcrypt unregistered [ 776.554527] Key type ._llcrypt registered [ 776.557087] Key type .llcrypt registered [ 777.996788] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:19:47 (1788776387) [ 778.366180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.381720] alg: No test for adler32 (adler32-zlib) [ 779.301994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.388774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.413829] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 780.172318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.199848] LNet: 40898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.208346] LNetError: 40898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.229472] LNet: Removed LNI 192.168.204.39@tcp [ 780.258687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.323890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.39@tcp [ 780.347511] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 781.042744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.39@tcp [ 781.076136] LNet: Removed LNI 192.168.204.39@tcp [ 781.146644] LNet: 41033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.655744] Key type .llcrypt unregistered [ 781.659990] Key type ._llcrypt unregistered [ 792.656455] Key type ._llcrypt registered [ 792.661625] Key type .llcrypt registered [ 794.103312] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:20:03 (1788776403) [ 795.906091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg439-server [ 797.412138] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:20:07 (1788776407) [ 798.057676] Key type .llcrypt unregistered [ 798.063562] Key type ._llcrypt unregistered [ 804.695707] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.978354] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.980689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 805.008122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.989562] Key type ._llcrypt registered [ 807.992748] Key type .llcrypt registered [ 808.213707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.241378] alg: No test for adler32 (adler32-zlib) [ 809.266096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.390645] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 810.501772] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 813.758906] LNet: 42863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.776876] LNetError: 42863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.786359] LNet: Removed LNI 192.168.204.39@tcp1 [ 813.803703] LNet: Removed LNI 10.1.2.3@tcp2 [ 814.573853] Key type .llcrypt unregistered [ 814.575603] Key type ._llcrypt unregistered [ 828.891112] Key type ._llcrypt registered [ 828.893745] Key type .llcrypt registered [ 830.346357] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:20:39 (1788776439) [ 830.726524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 830.741580] alg: No test for adler32 (adler32-zlib) [ 831.738349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.813821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.847436] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 832.614663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.662848] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.671048] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.682699] LNet: Removed LNI 192.168.204.39@tcp [ 832.707294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.782457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 832.805701] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 838.665445] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:20:48 (1788776448) [ 838.879807] LNet: 44422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.884518] LNetError: 44422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.904427] LNet: Removed LNI 192.168.204.39@tcp [ 839.423447] Key type .llcrypt unregistered [ 839.426310] Key type ._llcrypt unregistered [ 844.118904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.148939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.152750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.171461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.771768] Key type ._llcrypt registered [ 846.776429] Key type .llcrypt registered [ 847.211454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.237674] alg: No test for adler32 (adler32-zlib) [ 848.161077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.223667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.246121] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 849.116928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 849.137826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.909990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.936952] LNet: 45161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.942940] LNetError: 45161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.951749] LNet: Removed LNI 10.1.2.3@tcp [ 850.961121] LNet: Removed LNI 192.168.204.39@tcp [ 850.977256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.042461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 851.072292] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 851.077746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 851.607987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.626646] LNet: 45189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.632972] LNetError: 45189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.638358] LNetError: 45189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 851.643289] LNet: Removed LNI 10.1.2.3@tcp [ 851.654611] LNet: Removed LNI 192.168.204.39@tcp [ 851.686327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.762675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 851.815976] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 851.824266] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.313814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 852.334855] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 853.376288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.406644] LNet: 45351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.421943] LNetError: 45351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.428260] LNetError: 45351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 853.433252] LNet: Removed LNI 10.1.2.3@tcp [ 853.436302] LNet: Removed LNI 192.168.204.39@tcp [ 853.447916] LNet: Removed LNI 192.168.204.39@tcp2 [ 853.472812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.530603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 853.550844] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 853.554446] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.558242] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 860.847161] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:21:10 (1788776470) [ 860.924614] LNet: 45940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.930835] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 860.937995] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 860.950557] LNet: Removed LNI 10.1.2.3@tcp [ 860.952745] LNet: Removed LNI 192.168.204.39@tcp [ 860.976558] LNet: Removed LNI 192.168.204.39@tcp2 [ 861.610481] Key type .llcrypt unregistered [ 861.613489] Key type ._llcrypt unregistered [ 868.424773] Key type ._llcrypt registered [ 868.427294] Key type .llcrypt registered [ 868.623753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.646236] alg: No test for adler32 (adler32-zlib) [ 869.728265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.837880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.908764] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 870.841098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.879521] LNet: 46340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.891573] LNetError: 46340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.912873] LNet: Removed LNI 192.168.204.39@tcp [ 870.961279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.040310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 871.096192] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 871.911655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.947726] LNet: 46371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.960739] LNetError: 46371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.991497] LNet: Removed LNI 192.168.204.39@tcp [ 872.043355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.135064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 872.193080] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 872.641320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.673356] LNet: Removed LNI 192.168.204.39@tcp [ 872.694576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.757562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 872.779173] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 873.164354] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.274065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.291151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.313060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.716770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.734579] LNet: 46479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.738937] LNet: 46479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 875.744983] LNetError: 46479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.750343] LNetError: 46479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 875.766104] LNet: Removed LNI 192.168.204.39@tcp [ 875.792118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.920980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 875.976942] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 883.189938] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:21:32 (1788776492) [ 883.323708] LNet: 47070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.333134] LNetError: 47070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.351036] LNet: Removed LNI 192.168.204.39@tcp [ 884.214131] Key type .llcrypt unregistered [ 884.216778] Key type ._llcrypt unregistered [ 890.342156] Key type ._llcrypt registered [ 890.345476] Key type .llcrypt registered [ 890.722203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.746630] alg: No test for adler32 (adler32-zlib) [ 891.658197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.760639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.809372] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 892.500184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.521766] LNet: 47472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.532678] LNetError: 47472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.550278] LNet: Removed LNI 192.168.204.39@tcp [ 892.583330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.669268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 892.724534] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 898.764251] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:21:48 (1788776508) [ 898.939619] LNet: 48064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.952894] LNetError: 48064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.970765] LNet: Removed LNI 192.168.204.39@tcp [ 899.524580] Key type .llcrypt unregistered [ 899.527169] Key type ._llcrypt unregistered [ 905.454656] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 905.565336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 905.569917] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 905.575780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 908.139975] Key type ._llcrypt registered [ 908.142831] Key type .llcrypt registered [ 908.335116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.351577] alg: No test for adler32 (adler32-zlib) [ 909.270201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.331507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.369054] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 910.124986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.141710] LNet: 48513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.147381] LNetError: 48513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.170953] LNet: Removed LNI 192.168.204.39@tcp [ 910.194273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.243500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 910.263475] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 916.691390] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:22:06 (1788776526) [ 916.789476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.810286] LNet: 49104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.817255] LNetError: 49104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.841229] LNet: Removed LNI 192.168.204.39@tcp [ 916.873739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.025996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 922.860801] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:22:12 (1788776532) [ 922.955648] LNet: 49684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.480160] Key type .llcrypt unregistered [ 923.483244] Key type ._llcrypt unregistered [ 929.428417] Key type ._llcrypt registered [ 929.431946] Key type .llcrypt registered [ 929.690206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.704402] alg: No test for adler32 (adler32-zlib) [ 930.682386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.761714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.802917] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 931.703209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.722940] LNet: 50087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.728586] LNetError: 50087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.736991] LNet: Removed LNI 192.168.204.39@tcp [ 931.754124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.816800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 931.839579] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 938.323118] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:22:28 (1788776548) [ 938.561417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.591328] LNet: 50680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.598626] LNetError: 50680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.616557] LNet: Removed LNI 192.168.204.39@tcp [ 938.639164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.757871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 945.750889] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:22:35 (1788776555) [ 947.886370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 949.834951] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:22:39 (1788776559) [ 951.218695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 953.433671] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:22:43 (1788776563) [ 955.216138] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 957.283664] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:22:46 (1788776566) [ 959.394154] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 960.949322] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:22:50 (1788776570) [ 962.737562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 964.285169] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:22:54 (1788776574) [ 964.528074] LNet: 52733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.097702] Key type .llcrypt unregistered [ 965.104530] Key type ._llcrypt unregistered [ 972.565936] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.871714] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.874782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.880123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 975.565803] Key type ._llcrypt registered [ 975.568634] Key type .llcrypt registered [ 975.813831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.828879] alg: No test for adler32 (adler32-zlib) [ 976.762715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.884150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 976.916550] LNet: Added LNI 192.168.204.39@tcp [8/512/0/180] [ 976.928113] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 976.934101] LNet: 53052:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 977.054947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.079924] LNet: 53071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.084473] LNetError: 53071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.088702] LNet: Removed LNI 10.1.2.3@tcp [ 977.090740] LNetError: 53071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.102463] LNet: Removed LNI 192.168.204.39@tcp [ 977.137684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.204244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.235411] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 977.294588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.313379] LNet: 53095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.323396] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.337176] LNet: Removed LNI 192.168.204.39@tcp [ 977.369332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.454449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.493489] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 977.498255] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.499974] LNet: 53103:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 977.619534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 977.658904] LNet: 53121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.668577] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 977.676282] LNet: Removed LNI 10.1.2.3@tcp [ 977.683726] LNetError: 53121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.713629] LNet: Removed LNI 192.168.204.39@tcp [ 977.781840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.904486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.937953] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 977.944652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 977.946555] LNet: 53129:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.174753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.195400] LNet: 53155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.200963] LNetError: 53155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.207745] LNet: Removed LNI 10.1.2.3@tcp [ 978.209782] LNetError: 53155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.233273] LNet: Removed LNI 192.168.204.39@tcp [ 978.254824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.344423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.377046] LNet: Added LNI 192.168.204.39@tcp [8/256/0/182] [ 978.384521] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 978.465913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.489856] LNet: 53183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.498836] LNetError: 53183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.506115] LNet: Removed LNI 10.1.2.3@tcp [ 978.507779] LNetError: 53183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.525338] LNet: Removed LNI 192.168.204.39@tcp [ 978.567419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.661906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.689833] LNet: Added LNI 192.168.204.39@tcp [8/256/0/182] [ 978.699248] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 978.701490] LNet: 53191:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.803892] LNet: Removed LNI 10.1.2.3@tcp [ 978.831070] LNet: Removed LNI 192.168.204.39@tcp [ 979.521977] Key type .llcrypt unregistered [ 979.529063] Key type ._llcrypt unregistered [ 991.567144] Key type ._llcrypt registered [ 991.572371] Key type .llcrypt registered [ 992.964909] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 06:23:22 (1788776602) [ 994.429159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 995.915477] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:23:25 (1788776605) [ 996.532819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 996.556810] alg: No test for adler32 (adler32-zlib) [ 997.455402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.519677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.549175] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 998.344250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.39 [ 998.362739] LNet: 54452:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 998.374505] LNet: Removed LNI 192.168.204.39@tcp [ 1004.996915] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:23:34 (1788776614) [ 1005.306428] LNet: 55022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.313709] LNetError: 55022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.334632] LNet: Removed LNI 192.168.204.39@tcp [ 1006.103798] Key type .llcrypt unregistered [ 1006.106309] Key type ._llcrypt unregistered [ 1013.111115] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.380141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.073896] Key type ._llcrypt registered [ 1016.076984] Key type .llcrypt registered [ 1016.252841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.268209] alg: No test for adler32 (adler32-zlib) [ 1017.615849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1025.382636] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:23:54 (1788776634) [ 1025.501629] LNet: 55923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.514719] LNetError: 55923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1025.553321] LNet: Removed LNI 10.1.2.3@tcp [ 1026.296236] Key type .llcrypt unregistered [ 1026.299302] Key type ._llcrypt unregistered [ 1032.891101] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.162509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.180811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.192453] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.190247] Key type ._llcrypt registered [ 1036.193740] Key type .llcrypt registered [ 1036.406685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.441574] alg: No test for adler32 (adler32-zlib) [ 1037.417610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.391657] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:24:13 (1788776653) [ 1045.777623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1047.654928] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:24:17 (1788776657) [ 1047.887484] LNet: 57108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.907556] LNetError: 57108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.968240] LNet: Removed LNI 10.1.2.3@tcp [ 1049.007932] Key type .llcrypt unregistered [ 1049.014111] Key type ._llcrypt unregistered [ 1055.772618] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.071357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1056.079093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1056.085243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.869943] Key type ._llcrypt registered [ 1058.874925] Key type .llcrypt registered [ 1059.067725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.091853] alg: No test for adler32 (adler32-zlib) [ 1060.040780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.129538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.991086] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:24:38 (1788776678) [ 1069.441655] LNet: 58343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.446971] LNetError: 58343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.461288] LNet: Removed LNI 10.1.2.3@tcp [ 1070.365511] Key type .llcrypt unregistered [ 1070.371777] Key type ._llcrypt unregistered [ 1077.315352] Key type ._llcrypt registered [ 1077.323866] Key type .llcrypt registered [ 1077.698233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.717826] alg: No test for adler32 (adler32-zlib) [ 1078.729785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1078.804251] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1087.186275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1087.958974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_interrupt -r 1 [ 1087.989646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_interrupt -r 1 [ 1088.040423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1088.066151] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1088.245131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.282599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.333136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1088.366660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1088.393736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1088.435953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1088.481702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1088.511821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1088.530103] LNet: 58779:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1088.548928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1088.593992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1088.626428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.652384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.699227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.740112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.773915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1089.211403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_dropped -r 1 [ 1089.253841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_dropped -r 1 [ 1089.296960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1089.312097] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1089.499602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.527697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.551278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.580986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.604277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.630972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.656754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.711855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.736876] LNet: 58888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.763211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.816222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.859189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.905610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.972394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1090.035782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1090.095839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.633748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_aborted -r 1 [ 1090.677304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_aborted -r 1 [ 1090.706843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1090.738645] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1090.941850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.971158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.994221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.025283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.072090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.104442] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1091.118722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.144381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.177969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.195474] LNet: 58997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.217475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.255901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.313960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.388300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.470986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.503368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.539825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1092.111572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_no_route -r 1 [ 1092.147410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_no_route -r 1 [ 1092.182630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1092.395715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.431408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.457928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.491377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.522772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.544871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.574316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.607899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.645710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.681394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.744944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.773811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.807239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.843397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.883969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.590851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_timeout -r 1 [ 1093.647560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_timeout -r 1 [ 1093.730464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1093.762816] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1093.773542] LNet: Skipped 2 previous similar messages [ 1094.060885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.094609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.124834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.164369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.199335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.221609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.249487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.301396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.329203] LNet: 59216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.338809] LNet: 59216:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1094.362477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.408505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.447455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.483833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.528536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.576357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.608919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1095.172313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_error -r 1 [ 1095.211307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_error -r 1 [ 1095.244302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1095.441705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.488162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1095.553616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.598540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.633659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.666152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.735788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.791399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.857156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.895382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.934995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.958350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.985826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.021563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.057567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1096.278976] LNet: 59366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.287197] LNetError: 59366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.307109] LNet: Removed LNI 192.168.204.39@tcp [ 1096.733932] Key type .llcrypt unregistered [ 1096.739430] Key type ._llcrypt unregistered [ 1114.127455] Key type ._llcrypt registered [ 1114.130884] Key type .llcrypt registered [ 1115.741645] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:25:25 (1788776725) [ 1116.519369] Key type .llcrypt unregistered [ 1116.523431] Key type ._llcrypt unregistered [ 1123.584149] Key type ._llcrypt registered [ 1123.586972] Key type .llcrypt registered [ 1123.821619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.861779] alg: No test for adler32 (adler32-zlib) [ 1124.765511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.823540] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1132.876760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1142.989369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1143.060526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1143.112641] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1144.958319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_interrupt -r 1 [ 1145.017821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_interrupt -r 1 [ 1145.055914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_interrupt -r 1 [ 1145.089854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_interrupt -r 1 [ 1145.139462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_interrupt -r 1 [ 1145.172705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_interrupt -r 1 [ 1145.239658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_interrupt -r 1 [ 1145.322841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_interrupt -r 1 [ 1145.376656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1145.395391] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1145.401911] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1145.625360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.662315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.697920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.732809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.768896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.805428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.848227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.877365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.895274] LNet: 60830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.913592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.938871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.972557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1146.013396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1146.064835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.120338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.149342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.715778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_dropped -r 1 [ 1146.752270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_dropped -r 1 [ 1146.799406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_dropped -r 1 [ 1146.850882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_dropped -r 1 [ 1146.917703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_dropped -r 1 [ 1146.983180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_dropped -r 1 [ 1147.042877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_dropped -r 1 [ 1147.104593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_dropped -r 1 [ 1147.165703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1147.205332] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1147.210815] LNet: Skipped 2 previous similar messages [ 1147.449505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.471604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1147.504566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.548449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.597200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.628098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.653909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.686870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.707669] LNet: 60951:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.724840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.748296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.769752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.795399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.833913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.871629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.917592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.614268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_aborted -r 1 [ 1148.718602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_aborted -r 1 [ 1148.795294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_aborted -r 1 [ 1148.879164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_aborted -r 1 [ 1148.961745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_aborted -r 1 [ 1149.017587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_aborted -r 1 [ 1149.060706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_aborted -r 1 [ 1149.110491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_aborted -r 1 [ 1149.179385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1149.211346] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1149.216794] LNet: Skipped 2 previous similar messages [ 1149.529560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.554564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.580206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.610561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.673177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.733623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.767937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.815913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.834251] LNet: 61073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.855459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.898706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.942353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.998560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1150.074180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.123347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.169807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1150.878952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_no_route -r 1 [ 1150.919506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_no_route -r 1 [ 1150.962281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_no_route -r 1 [ 1150.992749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_no_route -r 1 [ 1151.024659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_no_route -r 1 [ 1151.053221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_no_route -r 1 [ 1151.082592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_no_route -r 1 [ 1151.123921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_no_route -r 1 [ 1151.199314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1151.233827] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1151.243076] LNet: Skipped 2 previous similar messages [ 1151.502628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.537636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.598905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.644313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.676820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.731244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.775256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.815438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.838051] LNet: 61194:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.881200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.939975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.998650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.049096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.109686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.180285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.227811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1152.790899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_timeout -r 1 [ 1152.831657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_timeout -r 1 [ 1152.861294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_timeout -r 1 [ 1152.898655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_timeout -r 1 [ 1152.954370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_timeout -r 1 [ 1152.993818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_timeout -r 1 [ 1153.045678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_timeout -r 1 [ 1153.110328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_timeout -r 1 [ 1153.143461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1153.339173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.364978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.388056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.408475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.429962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.454589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.478725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.510026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.557953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.583760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.616294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.642570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.662812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.695527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.716739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1154.234411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_error -r 1 [ 1154.269474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_error -r 1 [ 1154.306741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_error -r 1 [ 1154.366478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_error -r 1 [ 1154.401710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_error -r 1 [ 1154.432795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_error -r 1 [ 1154.467397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_error -r 1 [ 1154.499587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_error -r 1 [ 1154.529059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1154.709236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.732483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.760007] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.782129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.808559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.830784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.856440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.884907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.912539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.942778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.970148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.995249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.020296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.046484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.073472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.641814] LNet: 61513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.648541] LNetError: 61513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.657451] LNet: Removed LNI 192.168.204.39@tcp [ 1160.670028] LNet: Removed LNI 192.168.204.39@tcp1 [ 1161.217904] Key type .llcrypt unregistered [ 1161.219273] Key type ._llcrypt unregistered [ 1178.685353] Key type ._llcrypt registered [ 1178.687837] Key type .llcrypt registered [ 1180.542671] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:26:29 (1788776789) [ 1181.479338] Key type .llcrypt unregistered [ 1181.483650] Key type ._llcrypt unregistered [ 1188.552375] Key type ._llcrypt registered [ 1188.555797] Key type .llcrypt registered [ 1188.928601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1188.946990] alg: No test for adler32 (adler32-zlib) [ 1189.959404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1190.000268] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1198.593517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1199.460448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_dropped -r 1 [ 1199.498701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_dropped -r 1 [ 1199.531358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1199.550285] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1199.856635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.898739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.957950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.990180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.029162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.061461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.125941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.193243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.220496] LNet: 62758:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.248525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.278886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.314921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.368840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.420579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.487308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.543728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.234841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_error -r 1 [ 1201.311128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_error -r 1 [ 1201.394506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1201.420513] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1201.647531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.695662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1201.742332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.780879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.822952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.855124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.881082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.918650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.936966] LNet: 62868:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.958730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.992908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1202.036637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1202.085736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1202.125741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1202.154572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1202.179734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1202.596601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_timeout -r 1 [ 1202.644738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_timeout -r 1 [ 1202.677871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1202.696453] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1202.872969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1202.900676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1202.931917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1202.972570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1203.028398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1203.076501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1203.126994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1203.181782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1203.201897] LNet: 62977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.223278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1203.252478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1203.289931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1203.332499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1203.384060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1203.453326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1203.513156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1203.794530] LNet: 63018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1203.799714] LNetError: 63018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.834163] LNet: Removed LNI 192.168.204.39@tcp [ 1204.207639] Key type .llcrypt unregistered [ 1204.211504] Key type ._llcrypt unregistered [ 1220.302798] Key type ._llcrypt registered [ 1220.305418] Key type .llcrypt registered [ 1221.695202] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:27:11 (1788776831) [ 1222.271587] Key type .llcrypt unregistered [ 1222.275321] Key type ._llcrypt unregistered [ 1226.878810] Key type ._llcrypt registered [ 1226.881701] Key type .llcrypt registered [ 1227.125384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.143340] alg: No test for adler32 (adler32-zlib) [ 1228.079519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.109220] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1236.985249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1248.116481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.141108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1248.156858] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1249.892765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_dropped -r 1 [ 1249.928703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e remote_dropped -r 1 [ 1249.997525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_dropped -r 1 [ 1250.051633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e remote_dropped -r 1 [ 1250.093470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e remote_dropped -r 1 [ 1250.123618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e remote_dropped -r 1 [ 1250.145536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e remote_dropped -r 1 [ 1250.194841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e remote_dropped -r 1 [ 1250.249512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1250.268770] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1250.278795] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [ 1250.464119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.490767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.521368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.555871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.589700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.623224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.655326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.687853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.703779] LNet: 64481:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1250.721234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.762486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.802509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.839243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.884679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.932697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.976243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1251.023944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.839673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_error -r 1 [ 1251.895429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e remote_error -r 1 [ 1251.969346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_error -r 1 [ 1252.007601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e remote_error -r 1 [ 1252.082376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e remote_error -r 1 [ 1252.121265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e remote_error -r 1 [ 1252.176917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e remote_error -r 1 [ 1252.232645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e remote_error -r 1 [ 1252.285562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1252.304264] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1252.317217] LNet: Skipped 2 previous similar messages [ 1252.552902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1252.582434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1252.607315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1252.636923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1252.664493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1252.688301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1252.710827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1252.740255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1252.754573] LNet: 64605:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1252.769434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.792451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.818878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.850394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.886922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.928742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.966763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.007160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.593031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_timeout -r 1 [ 1253.684403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e remote_timeout -r 1 [ 1253.760146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_timeout -r 1 [ 1253.794125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e remote_timeout -r 1 [ 1253.824977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e remote_timeout -r 1 [ 1253.864547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e remote_timeout -r 1 [ 1253.929955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e remote_timeout -r 1 [ 1253.983951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e remote_timeout -r 1 [ 1254.038701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.139@tcp [ 1254.067954] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1254.351090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.401477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.440408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.468148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.501747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.530559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.576603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.609960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.635446] LNet: 64728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.651238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.709112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.740603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.779590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1254.802818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.826452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1254.851862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1254.884202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.785460] LNet: 64807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.801439] LNetError: 64807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1261.813624] LNet: Removed LNI 192.168.204.39@tcp [ 1261.823840] LNet: Removed LNI 192.168.204.39@tcp1 [ 1262.408604] Key type .llcrypt unregistered [ 1262.410713] Key type ._llcrypt unregistered [ 1283.125849] Key type ._llcrypt registered [ 1283.130102] Key type .llcrypt registered [ 1284.662835] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:28:14 (1788776894) [ 1286.658799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1288.490951] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:28:18 (1788776898) [ 1289.483684] Key type .llcrypt unregistered [ 1289.491277] Key type ._llcrypt unregistered [ 1295.519768] Key type ._llcrypt registered [ 1295.523828] Key type .llcrypt registered [ 1295.829457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.860130] alg: No test for adler32 (adler32-zlib) [ 1296.765651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.785779] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1306.010754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1306.871713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e network_timeout -r 1 [ 1306.928151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e network_timeout -r 1 [ 1306.968469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1306.984712] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1307.041306] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1307.255573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.283773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.333290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.390972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.435791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.483748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.540116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.585972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.609563] LNet: 66350:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.649903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.719628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.779104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.815294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.863544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.928650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.330589] LNet: 66389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.350471] LNetError: 66389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.385791] LNet: Removed LNI 192.168.204.39@tcp [ 1308.907448] Key type .llcrypt unregistered [ 1308.909206] Key type ._llcrypt unregistered [ 1328.235991] Key type ._llcrypt registered [ 1328.238268] Key type .llcrypt registered [ 1328.461060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.477204] alg: No test for adler32 (adler32-zlib) [ 1329.413405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.433072] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1337.406651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1347.033341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.085329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1347.127351] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1348.810933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e network_timeout -r 1 [ 1348.852877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e network_timeout -r 1 [ 1348.930156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e network_timeout -r 1 [ 1348.982816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e network_timeout -r 1 [ 1349.038862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e network_timeout -r 1 [ 1349.099709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e network_timeout -r 1 [ 1349.145665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e network_timeout -r 1 [ 1349.181974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e network_timeout -r 1 [ 1349.211435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1349.232506] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1349.463866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1349.505041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1349.557399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1349.603769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1349.635223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.679417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.713728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.742204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.766749] LNet: 67263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.785716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.823538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.853726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.882503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.917780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.920134] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1349.930917] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [ 1349.966242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.217837] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1355.675967] LNet: 67337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.691455] LNetError: 67337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.703937] LNet: Removed LNI 192.168.204.39@tcp [ 1355.726761] LNet: Removed LNI 192.168.204.39@tcp1 [ 1356.273654] Key type .llcrypt unregistered [ 1356.275738] Key type ._llcrypt unregistered [ 1374.441090] Key type ._llcrypt registered [ 1374.442571] Key type .llcrypt registered [ 1375.950665] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:29:45 (1788776985) [ 1376.387389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.433129] alg: No test for adler32 (adler32-zlib) [ 1377.479558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.546159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1377.571995] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1378.441474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1378.472578] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1379.430315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 1379.517211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1379.573648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1379.637824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1379.674882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1379.712836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1379.752924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.39@tcp [ 1379.775723] LNetError: 68435:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.39@tcp to recovery [ 1380.577941] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1380.578640] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1382.624377] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1383.410627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1386.720620] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1387.035405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1387.076191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1387.153618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.186401] LNet: 68543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.192657] LNetError: 68543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.258513] LNet: Removed LNI 192.168.204.39@tcp [ 1388.263640] LNetError: 68543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.284712] LNet: Removed LNI 192.168.204.39@tcp1 [ 1388.331554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.381963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.408768] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1389.343687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.375691] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1390.480430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 1390.525181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1390.619837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1390.667796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1390.729513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1390.788980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1390.836802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1390.919945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.39@tcp [ 1390.959680] LNetError: 68815:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.39@tcp to recovery [ 1391.456551] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1394.697960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1397.601062] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1397.617318] LNet: Skipped 1 previous similar message [ 1401.953709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1401.983727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1402.018694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1409.651854] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:30:19 (1788777019) [ 1409.936170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.977093] LNet: 69523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.998920] LNetError: 69523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.006213] LNet: Removed LNI 192.168.204.39@tcp [ 1410.027696] LNet: Removed LNI 192.168.204.39@tcp1 [ 1410.076706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.148377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.169519] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1411.302925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.330265] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1412.542995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 1412.648469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1412.677436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1412.707452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1412.759145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1412.786403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.813593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.39@tcp --health 0 [ 1412.883181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.219759] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.39@tcp [ 1413.220582] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1413.241492] LNet: Skipped 1 previous similar message [ 1428.276211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.424506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.460886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1428.507613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.39@tcp --health 500 [ 1428.599806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.294391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1435.373655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.408042] LNet: 70014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.424919] LNetError: 70014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.435837] LNetError: 70014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1435.447244] LNet: Removed LNI 192.168.204.39@tcp [ 1435.477957] LNet: Removed LNI 192.168.204.39@tcp1 [ 1435.513393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.603862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.632640] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1436.490288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.508840] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1437.248421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 1437.280663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.340628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1437.365800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1437.405254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1437.447646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1437.498319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1437.555773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.39@tcp --health 0 [ 1438.624474] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1438.631306] LNet: Skipped 3 previous similar messages [ 1438.937308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1449.218439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1449.260948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1449.293430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1457.191906] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:31:06 (1788777066) [ 1457.631320] LNet: 71061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.641965] LNetError: 71061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.651786] LNetError: 71061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1457.663470] LNet: Removed LNI 192.168.204.39@tcp [ 1457.698949] LNet: Removed LNI 192.168.204.39@tcp1 [ 1458.590660] Key type .llcrypt unregistered [ 1458.597144] Key type ._llcrypt unregistered [ 1465.916430] Key type ._llcrypt registered [ 1465.920355] Key type .llcrypt registered [ 1466.272722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.292775] alg: No test for adler32 (adler32-zlib) [ 1467.300219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1467.339635] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1476.888405] Lustre: DEBUG MARKER: Initial discovery [ 1477.129286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.139@tcp [ 1480.385679] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1480.627362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1480.704801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.139@tcp [ 1480.736664] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1480.772177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1480.835828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.139@tcp -r 1 -m GET -l 3 [ 1480.947529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.139@tcp -r 1 -m GET -e local_error [ 1481.029209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.139@tcp -r 1 -m PUT -l 6 [ 1481.697359] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1483.139530] Lustre: DEBUG MARKER: Force oleg439-client.virtnet to discover 192.168.204.139@tcp (in background) [ 1483.331314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.139@tcp [ 1485.580814] Lustre: DEBUG MARKER: Force oleg439-server to discover 192.168.204.39@tcp [ 1486.432807] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1488.508721] Lustre: DEBUG MARKER: Wait for 71817 [ 1489.570059] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1490.173471] Lustre: DEBUG MARKER: Finished wait on 71817 [ 1490.590692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1490.658068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1490.971522] LNet: 72272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.978932] LNetError: 72272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.009143] LNet: Removed LNI 192.168.204.39@tcp [ 1492.476554] Key type .llcrypt unregistered [ 1492.480233] Key type ._llcrypt unregistered [ 1510.283520] Key type ._llcrypt registered [ 1510.287589] Key type .llcrypt registered [ 1511.814090] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:32:01 (1788777121) [ 1512.544333] Key type .llcrypt unregistered [ 1512.546570] Key type ._llcrypt unregistered [ 1520.248796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1520.411475] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1520.419348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1520.456069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1523.109479] Key type ._llcrypt registered [ 1523.111098] Key type .llcrypt registered [ 1523.240696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1523.253887] alg: No test for adler32 (adler32-zlib) [ 1524.188527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.266785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.295558] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1525.355928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1525.406408] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1534.972319] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:32:24 (1788777144) [ 1535.257452] LNet: 74349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.268064] LNetError: 74349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1535.281680] LNet: Removed LNI 10.1.2.3@tcp [ 1535.300212] LNet: Removed LNI 192.168.204.39@tcp [ 1535.952736] Key type .llcrypt unregistered [ 1535.956816] Key type ._llcrypt unregistered [ 1543.947485] Key type ._llcrypt registered [ 1543.953673] Key type .llcrypt registered [ 1544.247477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1544.260974] alg: No test for adler32 (adler32-zlib) [ 1545.281609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.341174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.364551] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1546.234284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1546.259171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1547.396923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.39@tcp [ 1548.512450] LNetError: Unexpected error -22 connecting to 192.168.204.39@tcp at host 192.168.204.39:988 [ 1554.102270] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:32:43 (1788777163) [ 1554.258181] LNet: 75472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.263985] LNetError: 75472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1554.270441] LNet: Removed LNI 10.1.2.3@tcp [ 1554.284424] LNet: Removed LNI 192.168.204.39@tcp [ 1554.956695] Key type .llcrypt unregistered [ 1554.959620] Key type ._llcrypt unregistered [ 1562.033485] Key type ._llcrypt registered [ 1562.035657] Key type .llcrypt registered [ 1562.208938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1562.223382] alg: No test for adler32 (adler32-zlib) [ 1563.254323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.316620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1563.361478] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1564.159959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1564.179818] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 1565.053218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.39@tcp1 --nid 192.168.204.39@tcp2 [ 1569.584642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp1 192.168.204.39@tcp2 [ 1569.615399] LNetError: 76311:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.39@tcp1 to 192.168.204.39@tcp2 - Route table may be misconfigured [ 1569.637442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp2 192.168.204.39@tcp1 [ 1576.901774] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:33:06 (1788777186) [ 1577.010550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.054961] LNet: 76878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.067959] LNetError: 76878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.077136] LNet: Removed LNI 192.168.204.39@tcp1 [ 1578.155388] LNet: Removed LNI 192.168.204.39@tcp2 [ 1578.207032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.281095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.318651] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1578.998834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1579.026837] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1579.933286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 1580.020271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.39@tcp [ 1580.060550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp1 192.168.204.39@tcp1 [ 1580.123754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.39@tcp -d 192.168.204.39@tcp -e network_timeout [ 1580.193598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -e network_timeout [ 1580.256210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -e network_timeout [ 1580.324844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -e network_timeout [ 1580.406087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1580.447235] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1580.487308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1580.601839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1580.737893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1581.344750] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1587.316419] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:33:17 (1788777197) [ 1587.554926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.590975] LNet: 77747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.602759] LNetError: 77747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.618318] LNetError: 77747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1587.627771] LNet: Removed LNI 192.168.204.39@tcp [ 1588.713445] LNet: Removed LNI 192.168.204.39@tcp1 [ 1588.750434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.857903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1589.112933] LNet: 77784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.488707] Key type .llcrypt unregistered [ 1589.493756] Key type ._llcrypt unregistered [ 1602.047587] Key type ._llcrypt registered [ 1602.050956] Key type .llcrypt registered [ 1603.562660] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:33:33 (1788777213) [ 1604.002525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.014916] alg: No test for adler32 (adler32-zlib) [ 1605.016949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.536671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1608.159950] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:33:38 (1788777218) [ 1608.363299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.384389] LNet: 78879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.398641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.442466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1608.465426] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1609.092902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1609.125197] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1610.158391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1610.196657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp1 [ 1610.231657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp1 [ 1617.056272] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:33:46 (1788777226) [ 1619.128632] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg439-server [ 1620.968214] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:33:50 (1788777230) [ 1623.008952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg439-server [ 1625.272793] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:33:54 (1788777234) [ 1627.734163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg439-server [ 1629.635026] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:33:59 (1788777239) [ 1631.714800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg439-server [ 1633.909761] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:34:03 (1788777243) [ 1635.882803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg439-server [ 1637.716639] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:34:07 (1788777247) [ 1640.385957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg439-server [ 1642.819530] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:34:12 (1788777252) [ 1645.337306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg439-server [ 1647.467524] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:34:16 (1788777256) [ 1650.142159] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg439-server [ 1651.968615] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:34:21 (1788777261) [ 1654.855479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg439-server [ 1657.265128] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:34:26 (1788777266) [ 1657.380214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.421176] LNet: 82753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.433700] LNetError: 82753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.443337] LNet: Removed LNI 192.168.204.39@tcp [ 1658.471291] LNet: Removed LNI 192.168.204.39@tcp1 [ 1658.542770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.646572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.670464] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1659.627520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1659.718865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1659.875661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.889152] LNet: 82905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.894304] LNetError: 82905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.899479] LNetError: 82905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1659.916670] LNet: Removed LNI 192.168.204.39@tcp [ 1659.956549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.000971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1660.016865] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1660.813270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1661.081581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.123396] LNetError: 83050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.153646] LNet: Removed LNI 192.168.204.39@tcp [ 1661.179498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.328958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.372934] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1662.469699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1662.650964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1663.095935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.161459] LNet: 83198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.176665] LNet: 83198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1663.202578] LNetError: 83198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.235286] LNet: Removed LNI 192.168.204.39@tcp [ 1663.290604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.397621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1663.442552] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1664.288090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1664.564267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.702722] LNet: Removed LNI 192.168.204.39@tcp [ 1665.778662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.863448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.930145] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1667.248397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1667.392778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1668.763699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.787061] LNet: 83499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.793940] LNet: 83499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1668.810434] LNetError: 83499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.814873] LNetError: 83499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1668.827914] LNet: Removed LNI 192.168.204.39@tcp [ 1668.856292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.902266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1668.923706] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1670.137075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1670.361320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.464761] LNet: Removed LNI 192.168.204.39@tcp [ 1671.498372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1671.540549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1671.580199] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1672.573582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1672.730104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1674.334237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.432903] LNet: Removed LNI 192.168.204.39@tcp [ 1675.484083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.600602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1675.627026] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1676.493648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1676.766131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.802531] LNet: Removed LNI 192.168.204.39@tcp [ 1677.855586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.925532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.953182] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1678.549568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1678.636445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1678.936666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.960620] LNet: 84094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.965678] LNet: 84094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1678.973351] LNetError: 84094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.980753] LNetError: 84094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1678.998457] LNet: Removed LNI 192.168.204.39@tcp [ 1679.041488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.101464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1679.128269] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1679.955464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1680.179559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.257642] LNet: Removed LNI 192.168.204.39@tcp [ 1681.327093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.422912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.444931] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1682.139816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1682.222994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1682.546446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.624046] LNet: Removed LNI 192.168.204.39@tcp [ 1683.672172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.750422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1683.798764] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1684.728633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1685.047846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.124358] LNet: Removed LNI 192.168.204.39@tcp [ 1686.195990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.283352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.313133] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1687.084338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1687.144350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.182072] LNet: Removed LNI 192.168.204.39@tcp [ 1687.215688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.314752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.347699] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1688.009470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1688.091772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.122092] LNet: Removed LNI 192.168.204.39@tcp [ 1688.144035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.223408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1688.313582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.367982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.463583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1688.627109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.681764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.760663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1688.881115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.936739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.986529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1689.006560] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1689.839222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.873616] LNet: Removed LNI 192.168.204.39@tcp [ 1689.922669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.021542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1698.196609] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:35:07 (1788777307) [ 1698.739684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.757726] LNet: 85554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.763741] LNet: 85554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1698.792688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.890442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1698.980764] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/180] [ 1700.568742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.605323] LNetError: 85692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.615800] LNetError: 85692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1700.633763] LNet: Removed LNI 192.168.204.39@tcp231 [ 1700.673955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.753930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.779209] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/180] [ 1700.787208] LNet: 85700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.502660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.598202] LNet: Removed LNI 192.168.204.39@tcp231 [ 1701.637032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1701.703965] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/180] [ 1704.259885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.335973] LNet: Removed LNI 192.168.204.39@tcp231 [ 1704.391861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.549915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1704.602130] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/0] [ 1704.607754] LNet: 85853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.906800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1706.010439] LNet: Removed LNI 192.168.204.39@tcp231 [ 1706.052828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1706.105535] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/0] [ 1707.662186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.718081] LNet: Removed LNI 192.168.204.39@tcp231 [ 1707.793868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.925785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1707.946329] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/60] [ 1707.955982] LNet: 86005:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1708.254745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1708.284798] LNet: Removed LNI 192.168.204.39@tcp231 [ 1708.308613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1708.350213] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/60] [ 1710.092410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.125462] LNet: Removed LNI 192.168.204.39@tcp231 [ 1710.146964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.203558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1710.240726] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/180] [ 1710.246493] LNet: 86157:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1710.857002] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1710.897768] LNet: Removed LNI 192.168.204.39@tcp231 [ 1710.922921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1710.977368] LNet: Added LNI 192.168.204.39@tcp231 [8/256/0/180] [ 1723.008375] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:35:31 (1788777331) [ 1723.259954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.357843] LNet: Removed LNI 192.168.204.39@tcp231 [ 1723.446410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.582719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1723.649475] LNet: Added LNI 192.168.204.39@tcp232 [8/256/0/180] [ 1725.161688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1725.255449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1725.312518] LNet: Removed LNI 192.168.204.39@tcp232 [ 1725.348874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1725.410283] LNet: Added LNI 192.168.204.39@tcp232 [8/256/0/180] [ 1726.428355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1726.476086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1726.519549] LNet: Removed LNI 192.168.204.39@tcp232 [ 1726.549740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1726.582929] LNet: Added LNI 192.168.204.39@tcp232 [8/256/0/180] [ 1727.634562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1727.715232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1727.766509] LNet: Removed LNI 192.168.204.39@tcp232 [ 1727.790699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1727.842221] LNet: Added LNI 192.168.204.39@tcp232 [8/256/0/180] [ 1728.572541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1728.615911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1728.661211] LNet: Removed LNI 192.168.204.39@tcp232 [ 1728.674831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1735.672645] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:35:45 (1788777345) [ 1736.107453] LNet: 87996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.112491] LNet: 87996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1736.776934] Key type .llcrypt unregistered [ 1736.781839] Key type ._llcrypt unregistered [ 1743.615550] Key type ._llcrypt registered [ 1743.621554] Key type .llcrypt registered [ 1743.832599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.878596] alg: No test for adler32 (adler32-zlib) [ 1744.795807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1744.826508] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1751.658963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1759.716381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.761649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1759.783356] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1761.160964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_interrupt -r 2 [ 1761.207819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_interrupt -r 2 [ 1761.271794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_interrupt -r 2 [ 1761.337417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_interrupt -r 2 [ 1761.397443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_interrupt -r 2 [ 1761.435301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_interrupt -r 2 [ 1761.476605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_interrupt -r 2 [ 1761.519857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_interrupt -r 2 [ 1761.583834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1761.607442] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1761.611991] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1761.913170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1761.940845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1761.965867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1761.990622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.020513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.051745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.093472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.130867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.149772] LNet: 88656:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1762.175791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.216639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.243941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.272396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.304505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.341515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.074930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_dropped -r 2 [ 1763.112596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_dropped -r 2 [ 1763.171862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_dropped -r 2 [ 1763.203393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_dropped -r 2 [ 1763.248520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_dropped -r 2 [ 1763.291525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_dropped -r 2 [ 1763.331844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_dropped -r 2 [ 1763.403716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_dropped -r 2 [ 1763.462701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1763.492543] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1763.795488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.817841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.846635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1763.888842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1763.950699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.012797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.055103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1764.114350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1764.132719] LNet: 88777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.156250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.189833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.220516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.253305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.290092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.327253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1765.037140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_aborted -r 2 [ 1765.075656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_aborted -r 2 [ 1765.115641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_aborted -r 2 [ 1765.143455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_aborted -r 2 [ 1765.172344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_aborted -r 2 [ 1765.201814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_aborted -r 2 [ 1765.250112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_aborted -r 2 [ 1765.297785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_aborted -r 2 [ 1765.356464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1765.395953] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1765.802493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.867198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.922505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.967637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.014210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.037156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.070419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.110449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.135331] LNet: 88897:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.171572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.218517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.266634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.328738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.376353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.426718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1767.071747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_no_route -r 2 [ 1767.128486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_no_route -r 2 [ 1767.210517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_no_route -r 2 [ 1767.268274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_no_route -r 2 [ 1767.331138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_no_route -r 2 [ 1767.372281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_no_route -r 2 [ 1767.407883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_no_route -r 2 [ 1767.444498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_no_route -r 2 [ 1767.484779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1767.503685] LNet: There was an unexpected network error while writing to 192.168.204.139: rc = -22 [ 1767.878802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.920096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.966970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1768.020403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1768.051244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1768.114334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1768.167901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1768.209228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1768.252607] LNet: 89017:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1768.280868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1768.318097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1768.364852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1768.434710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.492661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.528779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1769.129617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e local_timeout -r 2 [ 1769.162245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e local_timeout -r 2 [ 1769.197271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e local_timeout -r 2 [ 1769.228620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e local_timeout -r 2 [ 1769.272112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e local_timeout -r 2 [ 1769.301533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e local_timeout -r 2 [ 1769.338607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e local_timeout -r 2 [ 1769.369683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e local_timeout -r 2 [ 1769.399113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1769.692465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.728505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.770118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.813686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.850649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.913940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.946334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.987091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1770.032803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1770.081892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1770.130329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.165796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.231630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.297105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1770.980665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp -m GET -e remote_dropped -r 2 [ 1771.020782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp -d 192.168.204.139@tcp1 -m GET -e remote_dropped -r 2 [ 1771.071858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp -m GET -e remote_dropped -r 2 [ 1771.122346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp -d 192.168.204.39@tcp1 -m GET -e remote_dropped -r 2 [ 1771.186990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp -m GET -e remote_dropped -r 2 [ 1771.224748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.139@tcp1 -m GET -e remote_dropped -r 2 [ 1771.275957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp -m GET -e remote_dropped -r 2 [ 1771.333110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.39@tcp1 -d 192.168.204.39@tcp1 -m GET -e remote_dropped -r 2 [ 1771.395577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.39@tcp 192.168.204.139@tcp [ 1771.418707] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [ 1771.657502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1771.699698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1771.739179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1771.785645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1771.824412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1771.863096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1771.905188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1771.953968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1771.990783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1772.026461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1772.064476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1772.094781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1772.118158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1772.147195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1777.880879] LNet: 89332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.896949] LNetError: 89332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.910253] LNet: Removed LNI 192.168.204.39@tcp [ 1777.924539] LNet: Removed LNI 192.168.204.39@tcp1 [ 1778.390569] Key type .llcrypt unregistered [ 1778.395302] Key type ._llcrypt unregistered [ 1793.714271] Key type ._llcrypt registered [ 1793.718034] Key type .llcrypt registered [ 1795.263634] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:36:44 (1788777404) [ 1795.700535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.719206] alg: No test for adler32 (adler32-zlib) [ 1796.613449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.709669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.735089] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1797.489739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1797.517810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.39@tcp [ 1797.536888] LNetError: 90309:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.39@tcp to recovery [ 1797.626283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.39@tcp [ 1797.732901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.603335] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:36:53 (1788777413) [ 1803.665583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.681771] LNet: 90892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.687869] LNetError: 90892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.699552] LNet: Removed LNI 192.168.204.39@tcp [ 1803.733129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.795219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.838193] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1804.875241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1804.905289] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 1805.895418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.39@tcp [ 1805.931425] LNetError: 91166:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.39@tcp to recovery [ 1806.816615] LNet: 1 local NIs in recovery (showing 1): 192.168.204.39@tcp [ 1806.819288] LNet: There was an unexpected network error while writing to 192.168.204.39: rc = -22 [ 1807.191364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.39@tcp [ 1815.620779] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:37:05 (1788777425) [ 1817.590229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg439-server [ 1819.121160] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:37:09 (1788777429) [ 1820.859499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg439-server [ 1822.769898] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:37:12 (1788777432) [ 1824.570565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg439-server [ 1826.357913] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:37:16 (1788777436) [ 1826.828671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.863289] LNet: 92796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.871432] LNetError: 92796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.882745] LNet: Removed LNI 192.168.204.39@tcp [ 1826.896992] LNet: Removed LNI 192.168.204.39@tcp1 [ 1826.977093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.070342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.111864] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1827.773631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1827.840564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1827.899678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1828.029722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1828.062651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1828.157891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1834.775683] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:37:24 (1788777444) [ 1835.267400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.288737] LNet: 93528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.301410] LNetError: 93528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.312534] LNetError: 93528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1835.330216] LNet: Removed LNI 192.168.204.39@tcp [ 1835.362203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.436138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.459488] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1843.791554] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:37:33 (1788777453) [ 1844.058987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.085818] LNet: 94231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.090948] LNetError: 94231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.110354] LNet: Removed LNI 192.168.204.39@tcp [ 1844.132284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.231070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.307479] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1851.584936] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:37:41 (1788777461) [ 1853.294484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1854.815944] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:37:44 (1788777464) [ 1855.245397] LNet: 95267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.254089] LNetError: 95267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.272185] LNet: Removed LNI 192.168.204.39@tcp [ 1855.852857] Key type .llcrypt unregistered [ 1855.858185] Key type ._llcrypt unregistered [ 1862.657322] Key type ._llcrypt registered [ 1862.660096] Key type .llcrypt registered [ 1862.825616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1862.846424] alg: No test for adler32 (adler32-zlib) [ 1863.802147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1863.828992] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1872.239875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 1877.465606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.139@tcp [ 1877.877169] LNet: 95682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.888349] LNetError: 95682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.915653] LNet: Removed LNI 192.168.204.39@tcp [ 1878.386737] Key type .llcrypt unregistered [ 1878.388694] Key type ._llcrypt unregistered [ 1890.460353] Key type ._llcrypt registered [ 1890.463551] Key type .llcrypt registered [ 1892.354499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1894.309264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1896.648161] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:38:26 (1788777506) [ 1898.471117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1900.721684] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:38:30 (1788777510) [ 1902.962710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg439-server [ 1905.230882] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:38:34 (1788777514) [ 1907.552787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg439-server [ 1909.758987] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:38:39 (1788777519) [ 1911.480534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1911.530396] alg: No test for adler32 (adler32-zlib) [ 1912.621562] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1912.898240] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1914.544847] Key type lgssc registered [ 1916.018072] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1937.629751] Key type lgssc unregistered [ 1937.932725] LNet: 98856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.936810] LNetError: 98856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.957667] LNet: Removed LNI 192.168.204.39@tcp [ 1938.668415] Key type .llcrypt unregistered [ 1938.672425] Key type ._llcrypt unregistered [ 1953.905234] Key type ._llcrypt registered [ 1953.909282] Key type .llcrypt registered [ 1955.213446] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:39:24 (1788777564) [ 1955.804675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.821379] alg: No test for adler32 (adler32-zlib) [ 1956.764373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.862376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1956.956827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1957.005554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1957.071436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1964.596613] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:39:34 (1788777574) [ 1964.966633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.995480] LNet: 100269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.021849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.102561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.144263] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1965.867155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.885408] LNet: 100413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.892986] LNetError: 100413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.907477] LNet: Removed LNI 192.168.204.39@tcp [ 1965.924718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.972366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1966.018656] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1966.697189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.734149] LNetError: 100441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.753879] LNet: Removed LNI 192.168.204.39@tcp [ 1966.774584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.842265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1966.880279] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1974.983256] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:39:44 (1788777584) [ 1975.134408] LNet: 101023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.145823] LNet: 101023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1975.169977] LNetError: 101023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.209245] LNet: Removed LNI 192.168.204.39@tcp [ 1976.084561] Key type .llcrypt unregistered [ 1976.089221] Key type ._llcrypt unregistered [ 1983.318512] Key type ._llcrypt registered [ 1983.321118] Key type .llcrypt registered [ 1983.642635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1983.664705] alg: No test for adler32 (adler32-zlib) [ 1984.701676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.806776] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 1984.812263] LNet: *** cfs_fail_loc=e003, val=100*** [ 1985.579224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 1985.602901] LNet: *** cfs_fail_loc=e003, val=100*** [ 1985.606580] LNet: Skipped 2 previous similar messages [ 1985.613245] Ping reply received with mapping: [ 1985.616932] found metadata reply NI[192.168.204.39@tcp] [ 1985.621962] 27 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1985.627802] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1985.631636] 5a 5a 5a 5a 5a 5a 5a 5a [ 1985.797229] LNet: 101428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.807572] LNetError: 101428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.839832] LNet: Removed LNI 192.168.204.39@tcp [ 1986.863545] Key type .llcrypt unregistered [ 1986.867340] Key type ._llcrypt unregistered [ 2000.387929] Key type ._llcrypt registered [ 2000.392886] Key type .llcrypt registered [ 2002.088627] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:40:11 (1788777611) [ 2002.743545] Key type .llcrypt unregistered [ 2002.748148] Key type ._llcrypt unregistered [ 2010.545208] Key type ._llcrypt registered [ 2010.547194] Key type .llcrypt registered [ 2010.700361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.723876] alg: No test for adler32 (adler32-zlib) [ 2011.892856] LNetError: 102466:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2012.285411] Key type .llcrypt unregistered [ 2012.289095] Key type ._llcrypt unregistered [ 2018.130537] Key type ._llcrypt registered [ 2018.133757] Key type .llcrypt registered [ 2019.825687] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:40:29 (1788777629) [ 2020.328590] Key type .llcrypt unregistered [ 2020.332411] Key type ._llcrypt unregistered [ 2026.952610] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2026.959515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2029.671259] Key type ._llcrypt registered [ 2029.673888] Key type .llcrypt registered [ 2029.983162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2030.008530] alg: No test for adler32 (adler32-zlib) [ 2031.005350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.097452] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2041.697465] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:40:51 (1788777651) [ 2042.097382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2042.139465] LNet: 104242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.147262] LNetError: 104242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2042.161627] LNet: Removed LNI 10.1.2.3@tcp [ 2042.202715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.067544] Key type .llcrypt unregistered [ 2043.072658] Key type ._llcrypt unregistered [ 2050.219297] Key type ._llcrypt registered [ 2050.222677] Key type .llcrypt registered [ 2050.521544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.551647] alg: No test for adler32 (adler32-zlib) [ 2051.610390] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2051.702679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2051.910962] LNet: 104528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.923710] LNetError: 104528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.957222] LNet: Removed LNI 192.168.204.39@tcp [ 2052.571485] Key type .llcrypt unregistered [ 2052.573781] Key type ._llcrypt unregistered [ 2065.616104] Key type ._llcrypt registered [ 2065.617972] Key type .llcrypt registered [ 2066.818552] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:41:16 (1788777676) [ 2067.988274] Key type .llcrypt unregistered [ 2067.990682] Key type ._llcrypt unregistered [ 2074.815363] Key type ._llcrypt registered [ 2074.818588] Key type .llcrypt registered [ 2075.317311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2075.343853] alg: No test for adler32 (adler32-zlib) [ 2083.767267] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:41:33 (1788777693) [ 2083.960656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.009296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.098511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.129320] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2084.895873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.933737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2084.967583] LNet: Removed LNI 192.168.204.39@tcp [ 2085.311522] LNet: 106267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.753629] Key type .llcrypt unregistered [ 2085.761238] Key type ._llcrypt unregistered [ 2098.699402] Key type ._llcrypt registered [ 2098.706457] Key type .llcrypt registered [ 2100.247554] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:41:49 (1788777709) [ 2100.680442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.692705] alg: No test for adler32 (adler32-zlib) [ 2101.592968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.648929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.677377] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2102.452139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2102.495683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.39@tcp [ 2102.514653] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2102.521380] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.526324] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 0 [ 2102.532345] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2102.535103] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2102.540710] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.547127] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 0 [ 2102.551271] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2102.553449] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2102.560379] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.564807] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 2152 [ 2102.571284] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2102.577121] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2102.581730] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2102.585943] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2102, 0, 2152 [ 2102.591947] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2109.859400] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:41:59 (1788777719) [ 2110.309500] LNet: 107805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.313796] LNetError: 107805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.333214] LNet: Removed LNI 192.168.204.39@tcp [ 2110.915691] Key type .llcrypt unregistered [ 2110.919270] Key type ._llcrypt unregistered [ 2117.473202] Key type ._llcrypt registered [ 2117.480067] Key type .llcrypt registered [ 2117.787858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2117.803685] alg: No test for adler32 (adler32-zlib) [ 2118.703509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2118.727730] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2125.735688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 2133.516182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.557637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2133.592665] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 2134.351635] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2134.539831] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2134.546957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2134.549417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2136.855629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2136.884278] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2147.255598] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:42:36 (1788777756) [ 2147.373470] LNet: 109304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2147.398409] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.409265] LNet: Removed LNI 192.168.204.39@tcp [ 2148.448272] LNet: Removed LNI 192.168.204.39@tcp1 [ 2148.453313] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2148.473973] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2148.513704] LNet: Removed LNI 10.1.2.3@tcp99 [ 2149.269874] Key type .llcrypt unregistered [ 2149.273133] Key type ._llcrypt unregistered [ 2155.334796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2155.343384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2157.793503] Key type ._llcrypt registered [ 2157.795782] Key type .llcrypt registered [ 2158.043624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2158.058323] alg: No test for adler32 (adler32-zlib) [ 2159.009242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.061898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.079872] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2159.738227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.758040] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.050353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2162.093863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.127736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2162.140567] LNetError: 110053:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2162.178775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.244614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2162.573762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.666481] LNet: 110073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.673636] LNetError: 110073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2162.683707] LNet: Removed LNI 10.1.2.3@tcp [ 2162.691675] LNet: Removed LNI 192.168.204.39@tcp [ 2163.325534] Key type .llcrypt unregistered [ 2163.328595] Key type ._llcrypt unregistered [ 2177.514867] Key type ._llcrypt registered [ 2177.521690] Key type .llcrypt registered [ 2179.456729] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:43:08 (1788777788) [ 2180.107798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.141135] alg: No test for adler32 (adler32-zlib) [ 2181.148370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.243851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.264812] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2182.254436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2182.349610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg439-client@tcp [ 2189.885100] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:43:19 (1788777799) [ 2190.049156] LNet: 111595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.061341] LNetError: 111595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.090275] LNet: Removed LNI 192.168.204.39@tcp [ 2190.764452] Key type .llcrypt unregistered [ 2190.766615] Key type ._llcrypt unregistered [ 2198.213574] Key type ._llcrypt registered [ 2198.215776] Key type .llcrypt registered [ 2198.346440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.366619] alg: No test for adler32 (adler32-zlib) [ 2199.284528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2199.307439] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2199.340368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2206.622878] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:43:36 (1788777816) [ 2208.666697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2210.537927] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:43:40 (1788777820) [ 2212.637853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2215.317877] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:43:44 (1788777824) [ 2215.508435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.528707] LNet: 113023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.534500] LNetError: 113023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.553957] LNet: Removed LNI 192.168.204.39@tcp [ 2215.587700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.645853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.669046] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2216.613709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2216.689506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2224.586295] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:43:54 (1788777834) [ 2224.848916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.864933] LNet: 113736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.881497] LNetError: 113736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.919065] LNet: Removed LNI 192.168.204.39@tcp [ 2224.951956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.020962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2225.060970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.866472] Key type .llcrypt unregistered [ 2225.875375] Key type ._llcrypt unregistered [ 2237.599272] Key type ._llcrypt registered [ 2237.606304] Key type .llcrypt registered [ 2239.089750] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:44:08 (1788777848) [ 2239.481923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2239.504551] alg: No test for adler32 (adler32-zlib) [ 2240.502693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.600836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.644697] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2241.586591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2241.651629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2248.615253] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:44:18 (1788777858) [ 2248.813218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.842342] LNet: 115289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.847855] LNetError: 115289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.903404] LNet: Removed LNI 192.168.204.39@tcp [ 2249.950246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.027858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2250.078691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2256.734710] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:44:26 (1788777866) [ 2257.064511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.082519] LNet: 115869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.123965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.173285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2257.521521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2257.562736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2257.637426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2258.141363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2258.189174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2258.273772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.293954] LNet: 115902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.336870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.416028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2258.848359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2258.894866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2258.983220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2259.556862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2259.591389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2266.789260] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:44:36 (1788777876) [ 2267.018615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.076698] LNet: 116500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.119203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.220941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.256981] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2268.260447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2268.295334] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2269.248485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2269.324481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2269.663803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.694231] LNetError: 116782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.699560] LNet: Removed LNI 192.168.204.39@tcp [ 2269.713417] LNet: Removed LNI 192.168.204.39@tcp2 [ 2269.743831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.813897] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2269.827831] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2269.833989] LNet: 116788:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.540074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2270.800790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.838043] LNetError: 116825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.847308] LNetError: 116825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2271.904174] LNet: Removed LNI 192.168.204.39@tcp [ 2271.927895] LNet: Removed LNI 192.168.204.39@tcp2 [ 2272.001324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.101338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.122328] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2272.914770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.934748] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2274.042985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2274.081957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2274.162767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2274.451944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.505695] LNet: 117110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.520666] LNet: 117110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2274.529733] LNetError: 117110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.537667] LNetError: 117110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2274.547354] LNet: Removed LNI 192.168.204.39@tcp [ 2275.562447] LNet: Removed LNI 192.168.204.39@tcp2 [ 2275.611635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.786811] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2275.805109] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2275.820609] LNet: 117116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2276.615391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2276.879928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.920499] LNet: Removed LNI 192.168.204.39@tcp [ 2277.998810] LNet: Removed LNI 192.168.204.39@tcp2 [ 2278.062579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.125477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2278.160877] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2279.188740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2279.210338] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2280.120689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2280.172408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2280.414409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.436434] LNetError: 117432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.442261] LNetError: 117432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2280.447642] LNet: Removed LNI 192.168.204.39@tcp [ 2280.458765] LNet: Removed LNI 192.168.204.39@tcp2 [ 2280.475304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.528914] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2280.533836] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2280.538139] LNet: 117438:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2281.027613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2281.209263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.272176] LNet: Removed LNI 192.168.204.39@tcp [ 2282.282835] LNet: Removed LNI 192.168.204.39@tcp2 [ 2282.347756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.435319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.458500] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2283.282224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2283.300087] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2284.075772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2284.108760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2284.168434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2284.369550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.387217] LNet: 117760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.392177] LNet: 117760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2285.408183] LNet: Removed LNI 192.168.204.39@tcp [ 2286.438808] LNet: Removed LNI 192.168.204.39@tcp2 [ 2286.465817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.557716] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2286.564688] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2286.574310] LNet: 117767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2287.332777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2287.542844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2287.561394] LNet: Removed LNI 192.168.204.39@tcp [ 2288.614806] LNet: Removed LNI 192.168.204.39@tcp2 [ 2288.671413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.778244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.843656] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2289.572324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2289.593826] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2290.543727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2290.578196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2290.641077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2291.074183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.097831] LNetError: 118100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.110311] LNetError: 118100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2292.128286] LNet: Removed LNI 192.168.204.39@tcp [ 2292.158278] LNet: Removed LNI 192.168.204.39@tcp2 [ 2292.214954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.328482] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2292.350674] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2292.370454] LNet: 118106:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2292.903562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2293.343457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.369703] LNet: Removed LNI 192.168.204.39@tcp [ 2294.400289] LNet: Removed LNI 192.168.204.39@tcp2 [ 2294.431942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.519969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.549884] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2295.374174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2295.396123] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2296.231267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2296.277037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2296.314587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2296.378539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2296.760170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.824556] LNet: Removed LNI 192.168.204.39@tcp [ 2298.861228] LNet: Removed LNI 192.168.204.39@tcp2 [ 2298.894266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.027543] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2299.045952] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2299.534763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2307.852533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2309.385724] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:45:19 (1788777919) [ 2309.498313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.533649] LNet: 119221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.546956] LNet: 119221:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2309.555922] LNetError: 119221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.562146] LNetError: 119221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2309.578062] LNet: Removed LNI 192.168.204.39@tcp [ 2310.633284] LNet: Removed LNI 192.168.204.39@tcp2 [ 2310.678549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.024498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.049698] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2311.764559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2311.791840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2312.800986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.827714] LNet: Removed LNI 10.1.2.3@tcp [ 2312.840423] LNet: Removed LNI 192.168.204.39@tcp [ 2312.874995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.998961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2313.040208] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2313.054775] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2313.572611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2313.599510] LNet: Removed LNI 192.168.204.39@tcp [ 2313.610659] LNet: Removed LNI 10.1.2.3@tcp [ 2321.817480] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 06:45:31 (1788777931) [ 2322.239738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.310140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.438563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.467927] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2323.461692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2323.522863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.556315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.597930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.651205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.699806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.743093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.787129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.860038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.920321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2323.991265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.39@tcp [ 2324.201936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2333.539592] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:45:42 (1788777942) [ 2333.809415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.897817] LNet: Removed LNI 192.168.204.39@tcp [ 2334.976642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.248615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2335.483588] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2335.507337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2335.518759] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2337.999939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.028951] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2338.830994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2338.854859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2341.250579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2341.306872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2341.417737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.39@tcp -m PUT -e local_timeout -r 1 [ 2341.521195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2341.604418] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2341.606866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2341.831859] LNetError: 120906:0:(lib-move.c:4801:lnet_parse()) 192.168.204.39@tcp, src 192.168.204.39@tcp: Dropping PUT (error -108 looking up sender) [ 2342.691448] LNet: 121366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.698096] LNet: 121366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2342.717551] LNetError: 121366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2342.726512] LNetError: 121366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2342.737931] LNet: Removed LNI 10.1.2.3@tcp [ 2343.801198] LNet: Removed LNI 192.168.204.39@tcp [ 2344.294755] Key type .llcrypt unregistered [ 2344.297865] Key type ._llcrypt unregistered [ 2359.377559] Key type ._llcrypt registered [ 2359.379819] Key type .llcrypt registered [ 2360.621827] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:46:10 (1788777970) [ 2361.005437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2361.028666] alg: No test for adler32 (adler32-zlib) [ 2361.973095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.432723] LNet: 122216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.171810] Key type .llcrypt unregistered [ 2363.175059] Key type ._llcrypt unregistered [ 2370.124565] Key type ._llcrypt registered [ 2370.127315] Key type .llcrypt registered [ 2370.286790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2370.298805] alg: No test for adler32 (adler32-zlib) [ 2371.301774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2371.333932] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2379.065299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 2379.268723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2380.093444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.139@tcp [ 2380.389667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2380.427334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2380.451366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2380.483036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2380.513399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2380.547266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2380.574073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2380.577256] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [ 2380.606631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2380.622365] LNet: 122663:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2380.637591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2380.665430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2380.693324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2380.728944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2380.756523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2380.791977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2381.546612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2381.803881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2381.860692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2381.896175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2381.948639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2382.003440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2382.030271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2382.057230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2382.085397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2382.099871] LNet: 122780:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2382.120576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2382.146891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2382.171930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2382.196162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2382.229378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2382.252962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2382.832496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.139@tcp [ 2383.093066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2383.140627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2383.188333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2383.228498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2383.255509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2383.280516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2383.305911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2383.326469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2383.340500] LNet: 122897:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2383.354935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2383.388215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2383.432155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2383.504063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2383.550963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2383.592314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2384.273825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2384.526680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2384.570758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2384.615260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2384.656826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2384.707986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2384.750980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2384.780453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2384.807808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2384.834542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2384.878597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2384.915725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2384.963645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2385.022396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2385.072653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2385.529876] LNet: 123061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.537188] LNetError: 123061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.559867] LNet: Removed LNI 192.168.204.39@tcp [ 2386.202860] Key type .llcrypt unregistered [ 2386.207179] Key type ._llcrypt unregistered [ 2407.168824] Key type ._llcrypt registered [ 2407.170357] Key type .llcrypt registered [ 2409.233957] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:46:58 (1788778018) [ 2409.927331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.961479] alg: No test for adler32 (adler32-zlib) [ 2410.913053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.615832] LNet: 123949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.600407] Key type .llcrypt unregistered [ 2412.602122] Key type ._llcrypt unregistered [ 2420.657836] Key type ._llcrypt registered [ 2420.662568] Key type .llcrypt registered [ 2420.999971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2421.012620] alg: No test for adler32 (adler32-zlib) [ 2421.988141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2422.019126] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2430.053881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.139@tcp [ 2440.591728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.674054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2440.721532] LNet: Added LNI 192.168.204.39@tcp1 [8/256/0/180] [ 2442.293708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2443.539689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.139@tcp [ 2444.015682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2444.053566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2444.115861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2444.154470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2444.222055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2444.277418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2444.343637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2444.400482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2444.438035] LNet: 124600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2444.493164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2444.561942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2444.593421] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [ 2444.633194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2444.704273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2444.766520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2444.849475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2445.862568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2446.253147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.326515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2446.391934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.449453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2446.498186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2446.564752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2446.645663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.729112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.763900] LNet: 124718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.817906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.901358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.940155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.994150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2447.033418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.097374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2448.076088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.139@tcp [ 2448.432825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.491588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.549661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.608919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.665946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.719521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.778739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.821363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.849831] LNet: 124835:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.883738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.946800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.999959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2449.065350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2449.126687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.186553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.285389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2450.718466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.760760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.812488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.840722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.893491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.966250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2451.040534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2451.120215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2451.193350] LNet: 124952:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2451.256711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2451.317513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2451.372909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2451.423901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2451.491245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2451.575243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2457.883192] LNet: 125033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2457.891126] LNetError: 125033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.901674] LNet: Removed LNI 192.168.204.39@tcp [ 2457.921692] LNet: Removed LNI 192.168.204.39@tcp1 [ 2458.407997] Key type .llcrypt unregistered [ 2458.411508] Key type ._llcrypt unregistered [ 2477.775815] Key type ._llcrypt registered [ 2477.781744] Key type .llcrypt registered [ 2479.546870] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 06:48:08 (1788778088) [ 2479.998498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.018838] alg: No test for adler32 (adler32-zlib) [ 2481.057996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.207604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2481.253482] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2482.231371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2482.254232] LNet: Added LNI 192.168.204.39@tcp2 [8/256/0/180] [ 2483.522492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.39@tcp [ 2483.633486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2484.705666] LNet: Removed LNI 192.168.204.39@tcp [ 2491.918938] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 06:48:21 (1788778101) [ 2493.734487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2495.588204] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:48:25 (1788778105) [ 2503.258760] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:48:33 (1788778113) [ 2510.225714] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:48:40 (1788778120) [ 2517.324283] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:48:47 (1788778127) [ 2524.123546] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:48:54 (1788778134) [ 2531.721023] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:49:01 (1788778141) [ 2538.322854] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:49:08 (1788778148) [ 2545.609904] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:49:15 (1788778155) [ 2552.748315] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:49:22 (1788778162) [ 2560.851324] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:49:30 (1788778170) [ 2567.580169] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:49:37 (1788778177) [ 2575.715536] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:49:45 (1788778185) [ 2584.355960] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:49:53 (1788778193) [ 2591.726975] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:50:01 (1788778201) [ 2600.452930] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:50:09 (1788778209) [ 2610.227777] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:50:19 (1788778219) [ 2619.484730] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:50:28 (1788778228) [ 2628.772816] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:50:38 (1788778238) [ 2628.915487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2628.955691] LNet: 136719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.965414] LNetError: 136719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.990374] LNet: Removed LNI 192.168.204.39@tcp2 [ 2629.014712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2629.169308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2629.191546] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2630.485375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2630.518065] LNet: 136868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2630.528392] LNetError: 136868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2630.543659] LNet: Removed LNI 192.168.204.39@tcp [ 2630.572875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2630.654259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2630.690210] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2638.781543] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:50:48 (1788778248) [ 2646.357246] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:50:55 (1788778255) [ 2654.901756] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:51:04 (1788778264) [ 2661.673353] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:51:11 (1788778271) [ 2668.055758] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:51:18 (1788778278) [ 2674.747411] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:51:24 (1788778284) [ 2681.337641] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:51:31 (1788778291) [ 2687.869944] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:51:37 (1788778297) [ 2694.756458] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:51:44 (1788778304) [ 2702.219994] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:51:51 (1788778311) [ 2711.153287] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:52:00 (1788778320) [ 2719.353396] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:52:08 (1788778328) [ 2728.572653] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2503 sec ========== 06:52:17 (1788778337) [ 2729.005939] LNet: 144333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2729.023079] LNetError: 144333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2729.038480] LNet: Removed LNI 192.168.204.39@tcp [ 2729.956937] Key type .llcrypt unregistered [ 2729.963305] Key type ._llcrypt unregistered [ 2743.722799] Key type ._llcrypt registered [ 2743.730847] Key type .llcrypt registered [ 2744.391759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2744.402687] alg: No test for adler32 (adler32-zlib) [ 2745.617585] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2745.986452] LNet: Added LNI 192.168.204.39@tcp [8/256/0/180] [ 2747.712204] Key type lgssc registered [ 2748.876906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2814.943668] Lustre: Mounted lustre-client - version 2.17.57_103_g80273de [ 2820.802794] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2829.598182] Lustre: Unmounted lustre-client [ 2859.550652] Key type lgssc unregistered [ 2859.979440] LNet: 147282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2859.994080] LNetError: 147282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.015810] LNet: Removed LNI 192.168.204.39@tcp [ 2860.741162] Key type .llcrypt unregistered [ 2860.744575] Key type ._llcrypt unregistered