[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456153599 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001020] APIC: Switch to symmetric I/O mode setup [ 0.003132] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008281] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.010016] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.011011] pid_max: default: 32768 minimum: 301 [ 0.012139] LSM: Security Framework initializing [ 0.013063] Yama: becoming mindful. [ 0.014052] SELinux: Initializing. [ 0.015095] *** VALIDATE selinux *** [ 0.023048] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028282] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032127] *** VALIDATE tmpfs *** [ 0.034010] *** VALIDATE proc *** [ 0.035297] *** VALIDATE cgroup *** [ 0.036019] *** VALIDATE cgroup2 *** [ 0.038181] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041033] Spectre V2 : User space: Vulnerable [ 0.042019] Speculative Store Bypass: Vulnerable [ 0.044657] debug: unmapping init [mem 0xffffffff99c59000-0xffffffff99c60fff] [ 0.046176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047759] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048097] ... version: 2 [ 0.049025] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052010] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055354] rcu: Hierarchical SRCU implementation. [ 0.057538] smp: Bringing up secondary CPUs ... [ 0.058543] x86: Booting SMP configuration: [ 0.059030] .... node #0, CPUs: #1 #2 #3 [ 0.062173] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.145067] node 0 deferred pages initialised in 76ms [ 0.148015] devtmpfs: initialized [ 0.149055] x86/mm: Memory block size: 128MB [ 0.151272] gcov: version magic: 0x41383552 [ 0.153172] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155269] pinctrl core: initialized pinctrl subsystem [ 0.156239] [ 0.157016] ************************************************************* [ 0.158018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159016] ** ** [ 0.160018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161018] ** ** [ 0.162015] ** This means that this kernel is built to expose internal ** [ 0.163013] ** IOMMU data structures, which may compromise security on ** [ 0.164015] ** your system. ** [ 0.165021] ** ** [ 0.166014] ** If you see this message and you are not debugging the ** [ 0.167013] ** kernel, report this immediately to your vendor! ** [ 0.168011] ** ** [ 0.169013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170017] ************************************************************* [ 0.171724] NET: Registered protocol family 16 [ 0.172597] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176032] cpuidle: using governor menu [ 0.178966] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181584] PCI: Using configuration type 1 for base access [ 0.183200] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193087] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196123] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199081] cryptd: max_cpu_qlen set to 1000 [ 0.202235] ACPI: Added _OSI(Module Device) [ 0.203013] ACPI: Added _OSI(Processor Device) [ 0.204018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.209042] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215470] ACPI: Interpreter enabled [ 0.217088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.219015] ACPI: Using IOAPIC for interrupt routing [ 0.220168] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223437] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246342] acpiphp: Slot [2] registered [ 0.247069] acpiphp: Slot [5] registered [ 0.248072] acpiphp: Slot [6] registered [ 0.249032] acpiphp: Slot [3] registered [ 0.249843] acpiphp: Slot [4] registered [ 0.251047] acpiphp: Slot [7] registered [ 0.252100] acpiphp: Slot [8] registered [ 0.253085] acpiphp: Slot [9] registered [ 0.253945] acpiphp: Slot [10] registered [ 0.255089] acpiphp: Slot [11] registered [ 0.256166] acpiphp: Slot [12] registered [ 0.257110] acpiphp: Slot [13] registered [ 0.259130] acpiphp: Slot [14] registered [ 0.260105] acpiphp: Slot [15] registered [ 0.262112] acpiphp: Slot [16] registered [ 0.263126] acpiphp: Slot [17] registered [ 0.264116] acpiphp: Slot [18] registered [ 0.266155] acpiphp: Slot [19] registered [ 0.267101] acpiphp: Slot [20] registered [ 0.269108] acpiphp: Slot [21] registered [ 0.270067] acpiphp: Slot [22] registered [ 0.271178] acpiphp: Slot [23] registered [ 0.273114] acpiphp: Slot [24] registered [ 0.274100] acpiphp: Slot [25] registered [ 0.275077] acpiphp: Slot [26] registered [ 0.275997] acpiphp: Slot [27] registered [ 0.277069] acpiphp: Slot [28] registered [ 0.278079] acpiphp: Slot [29] registered [ 0.279066] acpiphp: Slot [30] registered [ 0.279982] acpiphp: Slot [31] registered [ 0.281053] PCI host bridge to bus 0000:00 [ 0.282020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.285028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.287034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.290029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300147] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303138] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306312] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.319561] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328529] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332080] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335414] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.349560] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.353022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.365030] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.369017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.386213] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.391013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.393015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.404022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.410000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.412320] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.415346] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.416202] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.418138] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.422149] iommu: Default domain type: Passthrough [ 0.424520] SCSI subsystem initialized [ 0.426133] ACPI: bus type USB registered [ 0.428094] usbcore: registered new interface driver usbfs [ 0.429054] usbcore: registered new interface driver hub [ 0.430105] usbcore: registered new device driver usb [ 0.431161] pps_core: LinuxPPS API ver. 1 registered [ 0.432007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.434059] PTP clock support registered [ 0.435138] EDAC MC: Ver: 3.0.0 [ 0.436409] PCI: Using ACPI for IRQ routing [ 0.437597] NetLabel: Initializing [ 0.438007] NetLabel: domain hash size = 128 [ 0.438985] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.440078] NetLabel: unlabeled traffic allowed by default [ 0.441131] vgaarb: loaded [ 0.442245] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.444011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.449904] clocksource: Switched to clocksource kvm-clock [ 0.548691] VFS: Disk quotas dquot_6.6.0 [ 0.550487] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.552528] *** VALIDATE ramfs *** [ 0.553166] *** VALIDATE hugetlbfs *** [ 0.554738] pnp: PnP ACPI init [ 0.556341] pnp: PnP ACPI: found 6 devices [ 0.574156] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.577611] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.579877] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.582340] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.584914] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.587453] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.590257] NET: Registered protocol family 2 [ 0.592840] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.597699] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.601042] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.605112] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.608109] TCP: Hash tables configured (established 65536 bind 65536) [ 0.610098] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.612333] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.614375] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.617120] NET: Registered protocol family 1 [ 0.619164] RPC: Registered named UNIX socket transport module. [ 0.620765] RPC: Registered udp transport module. [ 0.622032] RPC: Registered tcp transport module. [ 0.623074] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.624669] NET: Registered protocol family 44 [ 0.626282] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.628342] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.630380] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.632584] PCI: CLS 0 bytes, default 64 [ 0.634293] Unpacking initramfs... [ 2.012458] debug: unmapping init [mem 0xffff91ddbcc64000-0xffff91ddbffcffff] [ 2.016433] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.018676] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.021507] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.532393] Initialise system trusted keyrings [ 2.533599] Key type blacklist registered [ 2.535280] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.544389] zbud: loaded [ 2.547547] *** VALIDATE nfs *** [ 2.548479] *** VALIDATE nfs4 *** [ 2.550083] pstore: using deflate compression [ 2.553551] Platform Keyring initialized [ 2.654918] NET: Registered protocol family 38 [ 2.656947] Key type asymmetric registered [ 2.658534] Asymmetric key parser 'x509' registered [ 2.660593] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.664052] io scheduler mq-deadline registered [ 2.665982] io scheduler kyber registered [ 2.667492] io scheduler bfq registered [ 2.669563] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.672597] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.675890] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.678715] ACPI: Power Button [PWRF] [ 2.683708] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.689890] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.698046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.727658] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.757150] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.763400] Non-volatile memory driver v1.3 [ 2.764648] Linux agpgart interface v0.103 [ 2.796845] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.799860] vda: detected capacity change from 0 to 76685312 [ 2.814636] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.817762] vdb: detected capacity change from 0 to 1073741824 [ 2.824505] libphy: Fixed MDIO Bus: probed [ 2.830820] usbcore: registered new interface driver usbserial_generic [ 2.832487] usbserial: USB Serial support registered for generic [ 2.834068] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.838384] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.840783] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.842555] mousedev: PS/2 mouse device common for all mice [ 2.845648] rtc_cmos 00:05: RTC can wake from S4 [ 2.848347] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.848551] rtc_cmos 00:05: registered as rtc0 [ 2.853046] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.855542] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.856677] intel_pstate: CPU model not supported [ 2.862691] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.868261] hid: raw HID events driver (C) Jiri Kosina [ 2.869599] usbcore: registered new interface driver usbhid [ 2.871188] usbhid: USB HID core driver [ 2.872797] drop_monitor: Initializing network drop monitor service [ 2.874846] Initializing XFRM netlink socket [ 2.876730] NET: Registered protocol family 10 [ 2.880388] Segment Routing with IPv6 [ 2.881780] NET: Registered protocol family 17 [ 2.884312] mpls_gso: MPLS GSO support [ 2.889376] RAS: Correctable Errors collector initialized. [ 2.892112] AVX version of gcm_enc/dec engaged. [ 2.894104] AES CTR mode by8 optimization enabled [ 2.962622] sched_clock: Marking stable (2962560216, 0)->(3902785344, -940225128) [ 2.965976] registered taskstats version 1 [ 2.967926] Loading compiled-in X.509 certificates [ 2.969841] zswap: loaded using pool lzo/zbud [ 2.996500] Key type big_key registered [ 3.009782] Key type encrypted registered [ 3.011463] ima: No TPM chip found, activating TPM-bypass! [ 3.013615] ima: Allocated hash algorithm: sha1 [ 3.014796] ima: No architecture policies found [ 3.016082] evm: Initialising EVM extended attributes: [ 3.017334] evm: security.selinux [ 3.018136] evm: security.ima [ 3.018779] evm: security.capability [ 3.019664] evm: HMAC attrs: 0x1 [ 3.021610] rtc_cmos 00:05: setting system clock to 2026-09-09 07:48:36 UTC (1788940116) [ 3.025912] debug: unmapping init [mem 0xffffffff9ac03000-0xffffffff9adfffff] [ 3.028379] debug: unmapping init [mem 0xffffffff99982000-0xffffffff99c58fff] [ 3.036123] Write protecting the kernel read-only data: 28672k [ 3.038673] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.040825] debug: unmapping init [mem 0xffffffff98914000-0xffffffff989fffff] [ 3.068230] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.075733] systemd[1]: Detected virtualization kvm. [ 3.078427] systemd[1]: Detected architecture x86-64. [ 3.079808] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.104841] systemd[1]: No hostname configured. [ 3.106658] systemd[1]: Set hostname to . [ 3.108883] random: systemd: uninitialized urandom read (16 bytes read) [ 3.111472] systemd[1]: Initializing machine ID from random generator. [ 3.236131] random: systemd: uninitialized urandom read (16 bytes read) [ 3.238838] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.243216] random: systemd: uninitialized urandom read (16 bytes read) [ 3.245817] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.253781] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.801590] device-mapper: uevent: version 1.0.3 [ 3.803268] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.472120] virtio_net virtio0 ens2: renamed from eth0 [ 4.546712] scsi host0: ata_piix [ 4.591596] scsi host1: ata_piix [ 4.593403] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.595397] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.410977] dracut-initqueue[ 8.413558] random: fast init done [579]: RTNETLINK answers: File exists [ 9.531414] random: crng init done [ 9.534520] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.051217] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.286402] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.579370] SELinux: Disabled at runtime. [ 11.640485] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.650919] systemd[1]: Detected virtualization kvm. [ 11.652540] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.138716] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.142561] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.147490] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.152625] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.156946] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.167738] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.172954] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice User and Session Slice. [ 12.240846] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.633783] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.943979] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.950959] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.060560] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.077642] EDAC sbridge: Ver: 1.1.2 [ 14.226326] Key type dns_resolver registered [ 14.521702] NFS: Registering the id_resolver key type [ 14.522947] Key type id_resolver registered [ 14.524446] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg138-client login: [ 46.308296] libcfs: loading out-of-tree module taints kernel. [ 46.333553] Key type ._llcrypt registered [ 46.335270] Key type .llcrypt registered [ 46.659553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 46.666900] alg: No test for adler32 (adler32-zlib) [ 47.718662] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 48.071244] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 49.727255] Key type lgssc registered [ 51.209304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 118.473195] hrtimer: interrupt took 2595888 ns [ 215.006580] Lustre: Mounted lustre-client - version 2.17.58_2_g930488d [ 220.848866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.702707] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing check_logdir /tmp/testlogs/ [ 240.608431] Lustre: lustre-OST0000-osc-ffff91de0555b000: disconnect after 24s idle [ 243.569681] Lustre: DEBUG MARKER: oleg138-client.virtnet: executing yml_node [ 250.401116] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 253.903637] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 257.793445] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 260.951894] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 9 03:52:51 EDT 2026 [ 283.756636] Lustre: Unmounted lustre-client [ 318.341265] Key type lgssc unregistered [ 318.819500] LNet: 8415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.844974] LNetError: 8415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.860694] LNet: Removed LNI 192.168.201.38@tcp [ 320.077190] Key type .llcrypt unregistered [ 320.079558] Key type ._llcrypt unregistered [ 350.827784] Key type ._llcrypt registered [ 350.829709] Key type .llcrypt registered [ 351.141782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 351.175813] alg: No test for adler32 (adler32-zlib) [ 352.214434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 352.335488] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 352.396905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 357.378816] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 357.474120] LNet: 9288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.482832] LNetError: 9288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 357.500383] LNet: Removed LNI 192.168.201.38@tcp [ 358.314478] Key type .llcrypt unregistered [ 358.320806] Key type ._llcrypt unregistered [ 365.685841] Key type ._llcrypt registered [ 365.689698] Key type .llcrypt registered [ 367.202657] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 03:54:39 (1788940479) [ 367.951875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 367.998553] alg: No test for adler32 (adler32-zlib) [ 368.964948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.063979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.099927] LNet: 9705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.138317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.238711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 369.275265] LNet: 9712:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 379.481538] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 03:54:50 (1788940490) [ 379.684434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.776546] LNet: 10284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.855471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.075398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 390.495836] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 03:55:01 (1788940501) [ 390.769502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.794091] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.893233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.058858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 400.973889] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 03:55:11 (1788940511) [ 401.149743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.200738] LNet: 11453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.262728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.550882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 411.783941] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 03:55:22 (1788940522) [ 411.902130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.920363] LNet: 12042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.949418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.146182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 413.085270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 420.873527] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 03:55:32 (1788940532) [ 421.227858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.270828] LNet: 12639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.303459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.406462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 422.067314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 422.802908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 431.761845] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 03:55:43 (1788940543) [ 431.963668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.067854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.163852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 441.220789] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 03:55:52 (1788940552) [ 441.284278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.392916] LNet: 13814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.398478] LNet: 13814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 441.457516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.637369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 441.676273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 442.794554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 442.869954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 444.067397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 444.139531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 444.677908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 444.723633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 445.139490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 445.234738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 445.827691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 445.855803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 446.535255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 446.586522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 446.975207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 447.010344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 447.457873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 447.509215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 448.108784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 448.141308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 448.441774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 448.488140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 449.093838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 449.208766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 449.897619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 449.945541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 463.326326] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 03:56:14 (1788940574) [ 463.875430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.969266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.201088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 464.334109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 472.935956] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 03:56:24 (1788940584) [ 473.372934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.420346] LNet: 15092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.431027] LNet: 15092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 473.498012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.665318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 473.758809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 481.702168] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 03:56:33 (1788940593) [ 481.900334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.947560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.996946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 482.036593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 488.870577] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 03:56:40 (1788940600) [ 488.979980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.055553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.186519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 489.212399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 496.310981] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 03:56:48 (1788940608) [ 496.461088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.542150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.606598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 496.629892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 503.575648] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 03:56:55 (1788940615) [ 503.697108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.747296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.834429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 503.873472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 510.637839] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 03:57:02 (1788940622) [ 510.786834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.831377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.931639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.506792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.054742] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 03:57:10 (1788940630) [ 519.143383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.206627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.390544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.627716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.492816] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 03:57:18 (1788940638) [ 526.616026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.682677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.889577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.229866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 533.576336] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 03:57:25 (1788940645) [ 533.808676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.848276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.239599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.511778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.558047] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 03:57:32 (1788940652) [ 540.734725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.752264] LNet: 20509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.760580] LNet: 20509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 540.789497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.839588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.879665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.206159] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 03:57:39 (1788940659) [ 547.361474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.448373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.575865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.666550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 554.719798] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 03:57:46 (1788940666) [ 554.880454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.951511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.048785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 563.682710] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 03:57:54 (1788940674) [ 563.802539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.877395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.975654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 570.202688] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 03:58:02 (1788940682) [ 570.311944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.415541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.532500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 570.992695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 578.386632] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 03:58:10 (1788940690) [ 578.549376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.600332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.727322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 578.759549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 590.071541] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 03:58:20 (1788940700) [ 590.328977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.451714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.700730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 590.754671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 599.320768] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 03:58:31 (1788940711) [ 599.569099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.654482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.743743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 599.811551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 607.265918] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 03:58:39 (1788940719) [ 607.530688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.583000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.669903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 607.734846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 615.678305] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 03:58:47 (1788940727) [ 615.764337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.808750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.890982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 615.975843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 616.022498] LNetError: 25826:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 616.094815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 616.144920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 623.195710] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 03:58:55 (1788940735) [ 623.279816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.342598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.484546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mXk8aQU3xm [ 623.592909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mXk8aQU3xm [ 623.697288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mXk8aQU3xm [ 631.645021] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 03:59:03 (1788940743) [ 631.956590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.006526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.058471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 632.091188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 632.142835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 638.731266] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 03:59:10 (1788940750) [ 639.113834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.165455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.243962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.271486] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 640.949484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 642.479832] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 644.415392] Key type lgssc registered [ 646.054139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 650.252390] LNetError: 27588:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.38@tcp99 has no available nets [ 653.038456] Key type lgssc unregistered [ 653.501263] LNetError: 28694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.516446] LNet: Removed LNI 192.168.201.38@tcp [ 654.033532] Key type .llcrypt unregistered [ 654.035532] Key type ._llcrypt unregistered [ 661.292728] Key type ._llcrypt registered [ 661.294873] Key type .llcrypt registered [ 662.754575] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 03:59:34 (1788940774) [ 663.488345] Key type .llcrypt unregistered [ 663.491188] Key type ._llcrypt unregistered [ 670.426845] Key type ._llcrypt registered [ 670.431338] Key type .llcrypt registered [ 670.853206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.897223] alg: No test for adler32 (adler32-zlib) [ 672.026754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.239821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 672.285727] LNet: 29604:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 672.298251] LNet: 29604:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 672.305580] LNet: 29604:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 672.329293] LNet: Message forwarding will be enabled in 2s [ 675.423181] LNet: Message forwarding enabled [ 675.970419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 675.989756] LNet: Message forwarding will stop in 2s to 6s [ 679.539815] LNet: Message forwarding disabled [ 680.705750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 680.761406] LNet: 29702:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 680.773638] LNet: 29702:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 680.784928] LNet: 29702:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 680.825260] LNet: Message forwarding will be enabled in 2s [ 681.105546] LNet: 29715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.074520] Key type .llcrypt unregistered [ 682.084417] Key type ._llcrypt unregistered [ 697.974619] Key type ._llcrypt registered [ 697.976164] Key type .llcrypt registered [ 699.491648] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:00:11 (1788940811) [ 699.962969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.997584] alg: No test for adler32 (adler32-zlib) [ 700.964967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.086747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 709.881706] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:00:21 (1788940821) [ 710.199062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.257649] LNet: 31116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.291339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.404210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 710.473659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 710.547286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 710.591704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 710.643309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 710.678770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 710.724567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 710.784649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 710.868159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 710.895817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 710.969787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 711.034019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 711.087811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 711.117272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 711.146986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 711.176575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 711.205765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 711.232963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 711.271243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 711.289483] LNetError: 31159:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 717.247990] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:00:29 (1788940829) [ 717.460666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.475358] LNet: 31730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.493695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.561637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.844186] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:00:36 (1788940836) [ 725.528173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg138-server [ 727.546787] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:00:39 (1788940839) [ 727.697822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.727811] LNet: 32658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.757103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.838178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.855909] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 729.088386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[39-43]@tcp [ 729.148578] LNet: 32794:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 729.199740] LNetError: 32665:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.39@tcp is being used as a gateway but routing feature is not turned on [ 729.213687] LNetError: 32665:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.39@tcp has gone from up to down [ 737.488585] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:00:49 (1788940849) [ 739.548925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg138-server [ 741.651391] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:00:53 (1788940853) [ 741.774020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.819976] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.836050] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.862508] LNet: Removed LNI 192.168.201.38@tcp [ 741.920424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.055910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.080614] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 743.126712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[39-44/2]@tcp [ 743.150875] LNet: 33851:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 743.165978] LNet: 33851:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 743.190996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[39-44/2]@tcp [ 743.213084] LNetError: 33737:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.39@tcp is being used as a gateway but routing feature is not turned on [ 743.225211] LNetError: 33737:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 743.231942] LNetError: 33737:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.39@tcp has gone from up to down [ 743.241183] LNetError: 33737:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 751.362657] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:01:03 (1788940863) [ 751.578321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.726864] LNet: 34455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.748710] LNetError: 34455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.789486] LNet: Removed LNI 192.168.201.38@tcp [ 751.836867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.951348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 752.074443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.142725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.201763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.253679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.289136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.370728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 752.511087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.573492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.694733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.832506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.910546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.070922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 753.153696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.193870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.295509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.392638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.419844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.465240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 753.544938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.653950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.769662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.937867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.001377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.102387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 754.241887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.276690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.388228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 754.507548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.561312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.621676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 754.673374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.712726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.831353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.979823] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:01:13 (1788940873) [ 763.898776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg138-server [ 765.956898] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:01:17 (1788940877) [ 767.763851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg138-server [ 769.675892] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:01:21 (1788940881) [ 769.799622] LNet: 35848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.805972] LNet: 35848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 770.706432] Key type .llcrypt unregistered [ 770.708845] Key type ._llcrypt unregistered [ 779.140969] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.347765] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.358350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.371606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.395939] Key type ._llcrypt registered [ 782.400413] Key type .llcrypt registered [ 782.662595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 782.679650] alg: No test for adler32 (adler32-zlib) [ 783.733525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.800608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 783.865211] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 784.914953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.938425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.146915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 787.162780] LNet: Removed LNI 10.1.2.3@tcp [ 787.290241] LNet: 36575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.297682] LNetError: 36575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.309976] LNet: Removed LNI 192.168.201.38@tcp [ 787.766543] Key type .llcrypt unregistered [ 787.768234] Key type ._llcrypt unregistered [ 802.923111] Key type ._llcrypt registered [ 802.926401] Key type .llcrypt registered [ 804.587933] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:01:56 (1788940916) [ 805.290756] Key type .llcrypt unregistered [ 805.303428] Key type ._llcrypt unregistered [ 814.304569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.613363] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.624543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.631233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.524611] Key type ._llcrypt registered [ 817.527017] Key type .llcrypt registered [ 817.826846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.846739] alg: No test for adler32 (adler32-zlib) [ 818.929895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.998907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.024389] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 820.081254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 820.116599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.860694] LNet: 38065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.866800] LNetError: 38065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 823.872052] LNet: Removed LNI 10.1.2.3@tcp [ 823.888444] LNet: Removed LNI 192.168.201.38@tcp [ 824.630561] Key type .llcrypt unregistered [ 824.634959] Key type ._llcrypt unregistered [ 839.775582] Key type ._llcrypt registered [ 839.776921] Key type .llcrypt registered [ 841.589036] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:02:32 (1788940952) [ 842.341581] Key type .llcrypt unregistered [ 842.343332] Key type ._llcrypt unregistered [ 849.870363] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.959892] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.969159] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.979226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.623485] Key type ._llcrypt registered [ 852.625524] Key type .llcrypt registered [ 852.834396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.849868] alg: No test for adler32 (adler32-zlib) [ 853.733664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.778395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 853.798297] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 855.613420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 855.697237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 857.651202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 857.669980] LNet: Removed LNI 10.1.2.3@tcp [ 857.687870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 857.719900] LNet: Removed LNI 10.1.2.31@tcp [ 857.743476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 857.782795] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.772324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 858.807567] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 859.675564] LNet: 39979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.684953] LNetError: 39979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 859.704501] LNet: Removed LNI 10.1.2.31@tcp [ 859.716653] LNet: Removed LNI 10.1.2.3@tcp [ 860.383332] Key type .llcrypt unregistered [ 860.385886] Key type ._llcrypt unregistered [ 877.600563] Key type ._llcrypt registered [ 877.602977] Key type .llcrypt registered [ 880.230969] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:03:10 (1788940990) [ 881.317782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.355880] alg: No test for adler32 (adler32-zlib) [ 882.467076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.577907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.657151] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 884.064354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.135032] LNet: 40930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.155737] LNetError: 40930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.184263] LNet: Removed LNI 192.168.201.38@tcp [ 884.229182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.354563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.38@tcp [ 884.409625] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 885.962794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.38@tcp [ 886.023687] LNet: Removed LNI 192.168.201.38@tcp [ 886.129462] LNet: 41065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.214502] Key type .llcrypt unregistered [ 887.221652] Key type ._llcrypt unregistered [ 901.771297] Key type ._llcrypt registered [ 901.773401] Key type .llcrypt registered [ 904.229289] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:03:35 (1788941015) [ 906.530338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg138-server [ 909.165497] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:03:40 (1788941020) [ 910.162325] Key type .llcrypt unregistered [ 910.166529] Key type ._llcrypt unregistered [ 918.363263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.524165] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.528663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.532872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.391437] Key type ._llcrypt registered [ 920.392869] Key type .llcrypt registered [ 920.669220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.694760] alg: No test for adler32 (adler32-zlib) [ 921.776954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.870825] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 923.225646] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 926.807983] LNet: 42891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.816836] LNetError: 42891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 926.829878] LNet: Removed LNI 192.168.201.38@tcp1 [ 926.849065] LNet: Removed LNI 10.1.2.3@tcp2 [ 927.626514] Key type .llcrypt unregistered [ 927.629470] Key type ._llcrypt unregistered [ 942.961864] Key type ._llcrypt registered [ 942.964790] Key type .llcrypt registered [ 944.570247] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:04:16 (1788941056) [ 945.251486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 945.265376] alg: No test for adler32 (adler32-zlib) [ 946.271988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.338168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.355420] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 947.423549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.445511] LNet: 43858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.450858] LNetError: 43858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.469649] LNet: Removed LNI 192.168.201.38@tcp [ 947.490852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.569821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 947.599754] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 956.713906] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:04:28 (1788941068) [ 957.113815] LNet: 44450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.124605] LNetError: 44450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.147331] LNet: Removed LNI 192.168.201.38@tcp [ 957.960522] Key type .llcrypt unregistered [ 957.964511] Key type ._llcrypt unregistered [ 965.654152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.683575] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.685719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.688269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 968.311156] Key type ._llcrypt registered [ 968.313621] Key type .llcrypt registered [ 968.593996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.608913] alg: No test for adler32 (adler32-zlib) [ 969.568033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.626619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.651462] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 970.456302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 970.486434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 972.695577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.711760] LNet: 45183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.715927] LNetError: 45183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 972.718361] LNet: Removed LNI 10.1.2.3@tcp [ 972.728185] LNet: Removed LNI 192.168.201.38@tcp [ 972.749672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.811641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 972.839962] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 972.844835] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 973.431492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.482065] LNet: 45210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.490032] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 973.501972] LNetError: 45210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 973.510082] LNet: Removed LNI 10.1.2.3@tcp [ 973.532500] LNet: Removed LNI 192.168.201.38@tcp [ 973.577921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.627945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 973.648519] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 973.652045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 974.315624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 974.355977] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 976.083484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.110569] LNet: 45372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.122719] LNetError: 45372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 976.137912] LNetError: 45372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 976.154360] LNet: Removed LNI 10.1.2.3@tcp [ 976.156641] LNet: Removed LNI 192.168.201.38@tcp [ 976.172439] LNet: Removed LNI 192.168.201.38@tcp2 [ 976.223904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.360793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 976.465060] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 976.472216] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 976.475467] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 990.373716] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:05:02 (1788941102) [ 990.849303] LNet: 45971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.868614] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.882291] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 990.899652] LNet: Removed LNI 10.1.2.3@tcp [ 990.903527] LNet: Removed LNI 192.168.201.38@tcp [ 990.924325] LNet: Removed LNI 192.168.201.38@tcp2 [ 992.048799] Key type .llcrypt unregistered [ 992.053697] Key type ._llcrypt unregistered [ 999.237987] Key type ._llcrypt registered [ 999.240839] Key type .llcrypt registered [ 999.444262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.461962] alg: No test for adler32 (adler32-zlib) [ 1000.501526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.556113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1000.598349] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1001.840959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1001.865389] LNet: 46363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.869514] LNetError: 46363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.885974] LNet: Removed LNI 192.168.201.38@tcp [ 1001.931935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.067506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1002.122967] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1003.263419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.299537] LNet: 46393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.306835] LNetError: 46393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.337617] LNet: Removed LNI 192.168.201.38@tcp [ 1003.433485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.625765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1003.693971] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1004.752258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.800350] LNet: 46423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.811707] LNetError: 46423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.833478] LNet: Removed LNI 192.168.201.38@tcp [ 1004.863724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.903288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1004.916887] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1005.265964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1005.432615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1005.437383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1005.446360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1008.023927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.055940] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.070372] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.094080] LNet: Removed LNI 192.168.201.38@tcp [ 1008.111343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.174561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1008.212806] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1017.138591] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:05:29 (1788941129) [ 1017.289494] LNet: 47096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.295033] LNetError: 47096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.327316] LNet: Removed LNI 192.168.201.38@tcp [ 1018.588991] Key type .llcrypt unregistered [ 1018.593788] Key type ._llcrypt unregistered [ 1027.548939] Key type ._llcrypt registered [ 1027.556648] Key type .llcrypt registered [ 1027.911359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.935717] alg: No test for adler32 (adler32-zlib) [ 1029.081685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.250876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1029.311115] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1030.717840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.744385] LNet: 47497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.751171] LNetError: 47497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.781612] LNet: Removed LNI 192.168.201.38@tcp [ 1030.808339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.931756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1030.957966] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1039.256428] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:05:51 (1788941151) [ 1039.394506] LNet: 48082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.411493] LNetError: 48082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.445815] LNet: Removed LNI 192.168.201.38@tcp [ 1040.250894] Key type .llcrypt unregistered [ 1040.263806] Key type ._llcrypt unregistered [ 1047.680615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.862620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1047.869428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1047.876900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.478741] Key type ._llcrypt registered [ 1049.481548] Key type .llcrypt registered [ 1049.713527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.745438] alg: No test for adler32 (adler32-zlib) [ 1050.761505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.858702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1050.913034] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1051.929849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.989352] LNet: 48528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.004958] LNetError: 48528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.038455] LNet: Removed LNI 192.168.201.38@tcp [ 1052.096535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.185473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1052.205543] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1060.235556] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:06:11 (1788941171) [ 1060.513553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.561436] LNet: 49127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.566902] LNetError: 49127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.590731] LNet: Removed LNI 192.168.201.38@tcp [ 1060.626944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.763581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1069.817488] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:06:21 (1788941181) [ 1069.896644] LNet: 49700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.301128] Key type .llcrypt unregistered [ 1071.307201] Key type ._llcrypt unregistered [ 1078.872775] Key type ._llcrypt registered [ 1078.875300] Key type .llcrypt registered [ 1079.109506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.149848] alg: No test for adler32 (adler32-zlib) [ 1080.069629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.125945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.152739] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1081.538371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.581497] LNet: 50104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.591739] LNetError: 50104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.620184] LNet: Removed LNI 192.168.201.38@tcp [ 1081.695162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.806210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1081.845574] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1095.087932] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:06:45 (1788941205) [ 1095.190342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1095.245759] LNet: 50689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.255441] LNetError: 50689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.298220] LNet: Removed LNI 192.168.201.38@tcp [ 1095.334038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.467738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1105.021902] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:06:56 (1788941216) [ 1107.444983] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1109.362431] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:07:01 (1788941221) [ 1111.416743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1113.494133] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:07:05 (1788941225) [ 1115.569836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1117.607401] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:07:09 (1788941229) [ 1120.118871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1122.823909] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:07:13 (1788941233) [ 1125.209389] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1128.024577] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:07:19 (1788941239) [ 1128.499721] LNet: 52753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.521644] Key type .llcrypt unregistered [ 1129.524062] Key type ._llcrypt unregistered [ 1137.890118] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1137.994377] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1138.001412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1138.009697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1140.864194] Key type ._llcrypt registered [ 1140.867394] Key type .llcrypt registered [ 1141.107444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.119195] alg: No test for adler32 (adler32-zlib) [ 1142.117758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.178924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1142.207487] LNet: Added LNI 192.168.201.38@tcp [8/512/0/180] [ 1142.213048] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1142.216905] LNet: 53072:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1142.371947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1142.401844] LNet: 53091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.409695] LNetError: 53091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1142.415587] LNet: Removed LNI 10.1.2.3@tcp [ 1142.417974] LNetError: 53091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.429784] LNet: Removed LNI 192.168.201.38@tcp [ 1142.452494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.505411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1142.525970] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1142.586096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1142.604287] LNet: 53115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.609729] LNetError: 53115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.628231] LNet: Removed LNI 192.168.201.38@tcp [ 1142.658655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.747441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1142.764780] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1142.768427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1142.770169] LNet: 53123:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1142.835849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1142.865462] LNet: 53141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.871463] LNetError: 53141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1142.874469] LNet: Removed LNI 10.1.2.3@tcp [ 1142.878914] LNetError: 53141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.888201] LNet: Removed LNI 192.168.201.38@tcp [ 1142.901639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.958118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1142.975653] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1142.979620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1142.982294] LNet: 53149:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1143.196677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1143.215517] LNet: 53175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.220673] LNetError: 53175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1143.231590] LNet: Removed LNI 10.1.2.3@tcp [ 1143.233964] LNetError: 53175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.244906] LNet: Removed LNI 192.168.201.38@tcp [ 1143.267590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1143.353407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1143.378590] LNet: Added LNI 192.168.201.38@tcp [8/256/0/182] [ 1143.384400] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1143.479594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1143.493381] LNet: 53203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1143.498867] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1143.504586] LNet: Removed LNI 10.1.2.3@tcp [ 1143.506543] LNetError: 53203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1143.527470] LNet: Removed LNI 192.168.201.38@tcp [ 1143.544511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1143.604715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1143.634190] LNet: Added LNI 192.168.201.38@tcp [8/256/0/182] [ 1143.640037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1143.642429] LNet: 53211:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1143.726846] LNet: Removed LNI 10.1.2.3@tcp [ 1143.746873] LNet: Removed LNI 192.168.201.38@tcp [ 1144.333839] Key type .llcrypt unregistered [ 1144.337647] Key type ._llcrypt unregistered [ 1157.441709] Key type ._llcrypt registered [ 1157.445796] Key type .llcrypt registered [ 1159.541864] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 04:07:50 (1788941270) [ 1162.491194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1165.025161] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:07:56 (1788941276) [ 1166.016631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1166.042134] alg: No test for adler32 (adler32-zlib) [ 1167.086297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.129931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1167.151983] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1168.530543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.38 [ 1168.570390] LNet: 54473:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1168.585569] LNet: Removed LNI 192.168.201.38@tcp [ 1178.088514] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:08:10 (1788941290) [ 1178.558599] LNet: 55043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.577064] LNetError: 55043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.606951] LNet: Removed LNI 192.168.201.38@tcp [ 1179.474792] Key type .llcrypt unregistered [ 1179.479091] Key type ._llcrypt unregistered [ 1187.451666] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1187.827545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1190.673590] Key type ._llcrypt registered [ 1190.679104] Key type .llcrypt registered [ 1190.837869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.848744] alg: No test for adler32 (adler32-zlib) [ 1192.050977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1199.254505] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:08:31 (1788941311) [ 1199.445763] LNet: 55944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.452158] LNetError: 55944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1199.482315] LNet: Removed LNI 10.1.2.3@tcp [ 1200.064178] Key type .llcrypt unregistered [ 1200.069213] Key type ._llcrypt unregistered [ 1206.525327] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1206.865454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1206.870655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1206.888131] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1209.842108] Key type ._llcrypt registered [ 1209.844248] Key type .llcrypt registered [ 1209.979056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1209.997615] alg: No test for adler32 (adler32-zlib) [ 1211.133097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1218.430205] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:08:50 (1788941330) [ 1221.136399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1225.021664] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:08:55 (1788941335) [ 1225.502575] LNet: 57132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.549376] LNetError: 57132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1226.171652] LNet: Removed LNI 10.1.2.3@tcp [ 1227.291442] Key type .llcrypt unregistered [ 1227.293763] Key type ._llcrypt unregistered [ 1239.077390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1239.388718] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1239.396382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1239.434239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1241.285504] Key type ._llcrypt registered [ 1241.288127] Key type .llcrypt registered [ 1241.565412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1241.597241] alg: No test for adler32 (adler32-zlib) [ 1242.610069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.696477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1254.316113] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:09:26 (1788941366) [ 1254.890569] LNet: 58353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.896230] LNetError: 58353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1254.913236] LNet: Removed LNI 10.1.2.3@tcp [ 1255.389448] Key type .llcrypt unregistered [ 1255.390932] Key type ._llcrypt unregistered [ 1261.931840] Key type ._llcrypt registered [ 1261.939533] Key type .llcrypt registered [ 1262.131624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.172222] alg: No test for adler32 (adler32-zlib) [ 1263.270270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1263.342948] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1272.219141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1273.082820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_interrupt -r 1 [ 1273.116505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_interrupt -r 1 [ 1273.222708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1273.304916] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1273.647263] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1273.722841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.763823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1273.849622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1273.922572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.974799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1274.025771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.101432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.172480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.225765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.281893] LNet: 58790:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.325579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.347629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.401247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.454544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.528547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.606172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.654607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.507852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_dropped -r 1 [ 1275.546993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_dropped -r 1 [ 1275.604671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1275.672067] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1276.030701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.091984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.159355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.204679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.295588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1276.341320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.372169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.398531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.431599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.447276] LNet: 58901:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.473823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.533656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.576727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.638428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.706512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.723033] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1276.776918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.894201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.559487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_aborted -r 1 [ 1277.592938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_aborted -r 1 [ 1277.632635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1277.898455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.926763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.969460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.019899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.062763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1278.127249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.189039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.255138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.308344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.330599] LNet: 59013:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.350425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.391702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.421537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.447977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.506770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.562239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.595734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1279.693713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_no_route -r 1 [ 1279.777612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_no_route -r 1 [ 1279.835171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1279.863215] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1279.874501] LNet: Skipped 1 previous similar message [ 1280.207729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.255457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.305348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.345431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.389875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1280.457651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.527636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.589860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.656554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.674557] LNet: 59124:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.709210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.771414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.819328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.862581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.936655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.006981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.065199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1281.598840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_timeout -r 1 [ 1281.642520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_timeout -r 1 [ 1281.698749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1282.191321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.236619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.317559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.389126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.459395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1282.518904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.575903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.622492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.672187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.712578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.759523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.791797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.832578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.871552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.932569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.986189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1283.882881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_error -r 1 [ 1283.915938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_error -r 1 [ 1283.955616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1283.989428] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1283.992476] LNet: Skipped 2 previous similar messages [ 1284.186669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.212908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1284.242162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.272959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.300779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1284.332572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.370418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.413864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.446599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.494175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.545783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.603623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.669908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.733149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.833252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.895968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1285.213771] LNet: 59388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.226811] LNetError: 59388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.247839] LNet: Removed LNI 192.168.201.38@tcp [ 1285.779496] Key type .llcrypt unregistered [ 1285.781119] Key type ._llcrypt unregistered [ 1305.575570] Key type ._llcrypt registered [ 1305.581158] Key type .llcrypt registered [ 1307.428402] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:10:18 (1788941418) [ 1308.485447] Key type .llcrypt unregistered [ 1308.488824] Key type ._llcrypt unregistered [ 1316.041271] Key type ._llcrypt registered [ 1316.051508] Key type .llcrypt registered [ 1316.439985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1316.490565] alg: No test for adler32 (adler32-zlib) [ 1317.459445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1317.476772] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1325.531984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1335.864385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.930274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1335.956205] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1337.399201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_interrupt -r 1 [ 1337.455976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_interrupt -r 1 [ 1337.508397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_interrupt -r 1 [ 1337.617909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_interrupt -r 1 [ 1337.679304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_interrupt -r 1 [ 1337.760938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_interrupt -r 1 [ 1337.826556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_interrupt -r 1 [ 1337.866440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_interrupt -r 1 [ 1337.920174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1337.944963] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1337.950931] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1338.193224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.230872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.284301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.338334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.373932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1338.401477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.448272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.497403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.544488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.584527] LNet: 60852:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.622508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.695575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.768203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.800766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.892820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.964110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.976837] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1338.988107] LNet: Skipped 3 previous similar messages [ 1339.034860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1339.667556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_dropped -r 1 [ 1339.737491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_dropped -r 1 [ 1339.844649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_dropped -r 1 [ 1339.913862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_dropped -r 1 [ 1339.961623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_dropped -r 1 [ 1340.012666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_dropped -r 1 [ 1340.125088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_dropped -r 1 [ 1340.210857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_dropped -r 1 [ 1340.281990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1340.339976] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1340.658165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.714980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.756265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.801976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.852990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1340.910981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.961278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.039603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.103782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.129478] LNet: 60976:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.161994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.200930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.253875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.328605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.383734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.456170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.527161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.338330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_aborted -r 1 [ 1342.400570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_aborted -r 1 [ 1342.454598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_aborted -r 1 [ 1342.493851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_aborted -r 1 [ 1342.546225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_aborted -r 1 [ 1342.608494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_aborted -r 1 [ 1342.666934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_aborted -r 1 [ 1342.720985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_aborted -r 1 [ 1342.747585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1342.769173] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1342.776416] LNet: Skipped 2 previous similar messages [ 1343.140921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1343.239578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1343.268608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1343.313428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.341443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1343.398749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.430210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.473480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.508818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.534873] LNet: 61099:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.571195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.633991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.753053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.797419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.833196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.862798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.909086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.716164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_no_route -r 1 [ 1344.788809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_no_route -r 1 [ 1344.865752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_no_route -r 1 [ 1344.946575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_no_route -r 1 [ 1345.069891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_no_route -r 1 [ 1345.150922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_no_route -r 1 [ 1345.221975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_no_route -r 1 [ 1345.282463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_no_route -r 1 [ 1345.383666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1345.885114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1345.926794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1346.019170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1346.068986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1346.123229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1346.206168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.276555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.330160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.398841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.428966] LNet: 61223:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.471720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.551320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.620889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1346.660939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.721179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1346.756310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1346.793041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1347.506660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_timeout -r 1 [ 1347.532396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_timeout -r 1 [ 1347.557894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_timeout -r 1 [ 1347.581871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_timeout -r 1 [ 1347.606753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_timeout -r 1 [ 1347.639769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_timeout -r 1 [ 1347.665740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_timeout -r 1 [ 1347.690140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_timeout -r 1 [ 1347.723153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1347.745627] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1347.750814] LNet: Skipped 5 previous similar messages [ 1347.971682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.017836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.043492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.070325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.104820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1348.132630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.163930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.194606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.224903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1348.252921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1348.286968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1348.379497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.451156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1348.486141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.538331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.572855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.277286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_error -r 1 [ 1349.315915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_error -r 1 [ 1349.367751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_error -r 1 [ 1349.415303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_error -r 1 [ 1349.459372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_error -r 1 [ 1349.523426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_error -r 1 [ 1349.582571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_error -r 1 [ 1349.637789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_error -r 1 [ 1349.693830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1350.072673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.131980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.203325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.277723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.345713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1350.402434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.443805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.475817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.521813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.543161] LNet: 61469:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.552768] LNet: 61469:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1350.577280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.641732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.673847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.714766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.747172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.790937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.845395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.410263] LNet: 61546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.428715] LNetError: 61546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.451088] LNet: Removed LNI 192.168.201.38@tcp [ 1359.532535] LNet: Removed LNI 192.168.201.38@tcp1 [ 1360.285962] Key type .llcrypt unregistered [ 1360.289927] Key type ._llcrypt unregistered [ 1381.158800] Key type ._llcrypt registered [ 1381.162728] Key type .llcrypt registered [ 1382.237803] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:11:34 (1788941494) [ 1383.517447] Key type .llcrypt unregistered [ 1383.519229] Key type ._llcrypt unregistered [ 1391.477416] Key type ._llcrypt registered [ 1391.490575] Key type .llcrypt registered [ 1391.828329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1391.865815] alg: No test for adler32 (adler32-zlib) [ 1392.944541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1392.972494] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1401.789082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1402.771913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_dropped -r 1 [ 1402.816338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_dropped -r 1 [ 1402.859393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1402.880365] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1403.148472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1403.195385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1403.223816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1403.271569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1403.313598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1403.370458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1403.438856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1403.468973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1403.509943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1403.548622] LNet: 62792:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1403.583887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1403.623714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1403.669265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1403.714635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1403.743824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1403.782656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1403.818444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1404.399372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_error -r 1 [ 1404.483985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_error -r 1 [ 1404.563430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1404.613670] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1405.008859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1405.048021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1405.095827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1405.170772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1405.242923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1405.285500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1405.368703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1405.438876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1405.482469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1405.500201] LNet: 62903:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1405.545430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1405.597507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1405.652843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.710644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.744592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.789711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.837420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1406.697121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_timeout -r 1 [ 1406.730962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_timeout -r 1 [ 1406.757648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1406.774408] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1406.963823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1407.004614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1407.068850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1407.121590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1407.179678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1407.215397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1407.240498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1407.268833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1407.290388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1407.302523] LNet: 63014:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1407.317286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1407.362568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1407.402693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1407.459312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1407.493242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1407.527956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1407.575940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1407.880566] LNet: 63056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.889475] LNetError: 63056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.906323] LNet: Removed LNI 192.168.201.38@tcp [ 1408.596548] Key type .llcrypt unregistered [ 1408.599323] Key type ._llcrypt unregistered [ 1430.248662] Key type ._llcrypt registered [ 1430.252963] Key type .llcrypt registered [ 1431.701265] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:12:23 (1788941543) [ 1432.628963] Key type .llcrypt unregistered [ 1432.630855] Key type ._llcrypt unregistered [ 1441.180838] Key type ._llcrypt registered [ 1441.183162] Key type .llcrypt registered [ 1441.341168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.359029] alg: No test for adler32 (adler32-zlib) [ 1442.299790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1442.340948] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1452.266525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1463.617686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.693962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.754939] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1465.932700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_dropped -r 1 [ 1466.013789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e remote_dropped -r 1 [ 1466.081740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_dropped -r 1 [ 1466.118678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e remote_dropped -r 1 [ 1466.185930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e remote_dropped -r 1 [ 1466.244894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e remote_dropped -r 1 [ 1466.300307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e remote_dropped -r 1 [ 1466.357167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e remote_dropped -r 1 [ 1466.407436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1466.434342] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1466.446582] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.138@tcp [ 1466.809969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1466.899496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1466.970314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1467.065988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1467.148094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1467.228764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1467.289281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1467.323525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1467.387727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1467.417992] LNet: 64521:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1467.467619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1467.498084] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1467.525029] LNet: Skipped 3 previous similar messages [ 1467.578283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1467.658503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1467.687856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1467.714224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1467.742831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1467.771393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1467.834538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1468.641263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_error -r 1 [ 1468.716027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e remote_error -r 1 [ 1468.786449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_error -r 1 [ 1468.862078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e remote_error -r 1 [ 1468.933419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e remote_error -r 1 [ 1469.010709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e remote_error -r 1 [ 1469.116539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e remote_error -r 1 [ 1469.181718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e remote_error -r 1 [ 1469.259903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1469.296980] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1469.625613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1469.646797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1469.672689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1469.696771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1469.727340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1469.806160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1469.879853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1469.958937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1469.998816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1470.029258] LNet: 64647:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1470.073135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1470.141973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1470.248333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1470.322580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1470.422967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1470.485316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1470.531792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1470.606803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1471.227863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_timeout -r 1 [ 1471.303048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e remote_timeout -r 1 [ 1471.362812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_timeout -r 1 [ 1471.440953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e remote_timeout -r 1 [ 1471.507447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e remote_timeout -r 1 [ 1471.548616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e remote_timeout -r 1 [ 1471.586113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e remote_timeout -r 1 [ 1471.637474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e remote_timeout -r 1 [ 1471.743394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.138@tcp [ 1471.793224] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1472.115981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1472.145331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1472.174489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1472.199517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1472.230877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1472.268172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1472.306997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1472.335760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1472.407995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1472.451483] LNet: 64772:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1472.482048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1472.552632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1472.607275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1472.650786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1472.711460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1472.756412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1472.816054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1472.866967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1478.914271] LNet: 64850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.916832] LNetError: 64850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.920240] LNet: Removed LNI 192.168.201.38@tcp [ 1478.925673] LNet: Removed LNI 192.168.201.38@tcp1 [ 1479.280508] Key type .llcrypt unregistered [ 1479.282590] Key type ._llcrypt unregistered [ 1500.133834] Key type ._llcrypt registered [ 1500.135322] Key type .llcrypt registered [ 1502.208630] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:13:33 (1788941613) [ 1504.200928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1506.144100] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:13:38 (1788941618) [ 1507.348398] Key type .llcrypt unregistered [ 1507.350546] Key type ._llcrypt unregistered [ 1514.600171] Key type ._llcrypt registered [ 1514.602127] Key type .llcrypt registered [ 1514.832055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1514.846974] alg: No test for adler32 (adler32-zlib) [ 1515.800737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1515.843991] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1524.095125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1524.887677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e network_timeout -r 1 [ 1524.927962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e network_timeout -r 1 [ 1524.982980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1524.998481] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1525.087161] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1525.373599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1525.410711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1525.461149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1525.494252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1525.544205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1525.590872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1525.636444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1525.697701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1525.737976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1525.759660] LNet: 66392:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1525.788884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1525.840884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1525.883200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1525.931746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.965653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.010167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.330596] LNet: 66431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.335824] LNetError: 66431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.361268] LNet: Removed LNI 192.168.201.38@tcp [ 1526.724672] Key type .llcrypt unregistered [ 1526.726337] Key type ._llcrypt unregistered [ 1546.864470] Key type ._llcrypt registered [ 1546.866913] Key type .llcrypt registered [ 1547.081224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1547.111628] alg: No test for adler32 (adler32-zlib) [ 1548.092815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1548.122404] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1555.879070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1565.625059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.676928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1565.707702] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1567.362940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e network_timeout -r 1 [ 1567.414642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e network_timeout -r 1 [ 1567.444385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e network_timeout -r 1 [ 1567.493393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e network_timeout -r 1 [ 1567.542599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e network_timeout -r 1 [ 1567.592935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e network_timeout -r 1 [ 1567.676485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e network_timeout -r 1 [ 1567.722555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e network_timeout -r 1 [ 1567.757996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1567.777887] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1568.021497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1568.050883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1568.077219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1568.103686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1568.128454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1568.152596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1568.181121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1568.205722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1568.229746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1568.245707] LNet: 67308:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1568.261339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1568.290969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1568.317234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1568.367289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1568.406815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1568.424230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1572.665898] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1572.709990] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1573.109434] LNet: 67381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.118462] LNetError: 67381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.130380] LNet: Removed LNI 192.168.201.38@tcp [ 1573.144857] LNet: Removed LNI 192.168.201.38@tcp1 [ 1573.600548] Key type .llcrypt unregistered [ 1573.601908] Key type ._llcrypt unregistered [ 1595.849940] Key type ._llcrypt registered [ 1595.853118] Key type .llcrypt registered [ 1597.606562] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:15:08 (1788941708) [ 1598.484553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.544329] alg: No test for adler32 (adler32-zlib) [ 1599.831280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.035680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.117041] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1601.971957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1602.014471] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1603.923729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 1604.037950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1604.080311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1604.111310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1604.180913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1604.210570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1604.244166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.38@tcp [ 1604.274868] LNetError: 68480:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.38@tcp to recovery [ 1605.087304] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1605.092923] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1607.136751] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1608.260739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1611.231546] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1612.268800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1612.312457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1612.371500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.386575] LNet: 68587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.409382] LNetError: 68587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.423466] LNet: Removed LNI 192.168.201.38@tcp [ 1612.449692] LNet: Removed LNI 192.168.201.38@tcp1 [ 1612.498702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.567598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.626645] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1613.693752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1613.727529] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1615.143241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 1615.201947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.317704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1615.369790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1615.464471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1615.522745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1615.585670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1615.682926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.38@tcp [ 1615.728293] LNetError: 68860:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.38@tcp to recovery [ 1616.671471] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1619.645419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1622.815884] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1622.829458] LNet: Skipped 1 previous similar message [ 1628.009718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1628.074569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1628.120412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1637.175509] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:15:48 (1788941748) [ 1637.317283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.351555] LNet: 69572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.358820] LNetError: 69572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.367870] LNetError: 69572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1637.377545] LNet: Removed LNI 192.168.201.38@tcp [ 1637.393258] LNet: Removed LNI 192.168.201.38@tcp1 [ 1637.439267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.550257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.568671] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1638.396768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1638.416663] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1639.314585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 1639.415624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1639.457742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1639.491620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1639.540314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1639.577524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1639.622539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.38@tcp --health 0 [ 1639.752520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1640.607423] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.38@tcp [ 1640.608847] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1640.625971] LNet: Skipped 1 previous similar message [ 1655.805107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1656.027377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1656.080845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1656.123746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.38@tcp --health 500 [ 1656.243837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1664.160580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1664.246225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.275209] LNet: 70078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.292795] LNetError: 70078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.295919] LNetError: 70078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.304763] LNet: Removed LNI 192.168.201.38@tcp [ 1664.321614] LNet: Removed LNI 192.168.201.38@tcp1 [ 1664.346893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.440167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.474730] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1665.154224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1665.192549] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1666.110058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 1666.153663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.224433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1666.253347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1666.304774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1666.378168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1666.447849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1666.511821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.38@tcp --health 0 [ 1667.491473] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1667.495863] LNet: Skipped 3 previous similar messages [ 1667.844638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1678.276406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1678.300754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1678.327437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1686.098624] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:16:38 (1788941798) [ 1686.476257] LNet: 71124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.483887] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.491755] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.499643] LNet: Removed LNI 192.168.201.38@tcp [ 1686.513883] LNet: Removed LNI 192.168.201.38@tcp1 [ 1687.526607] Key type .llcrypt unregistered [ 1687.530140] Key type ._llcrypt unregistered [ 1694.811258] Key type ._llcrypt registered [ 1694.817732] Key type .llcrypt registered [ 1695.162605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1695.187705] alg: No test for adler32 (adler32-zlib) [ 1696.185325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1696.216195] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1707.934780] Lustre: DEBUG MARKER: Initial discovery [ 1708.040862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.138@tcp [ 1711.562160] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1711.685469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1711.782506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.138@tcp [ 1711.837024] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1711.860879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1711.954394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.138@tcp -r 1 -m GET -l 3 [ 1712.053452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.138@tcp -r 1 -m GET -e local_error [ 1712.134754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.138@tcp -r 1 -m PUT -l 6 [ 1712.611123] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1714.049683] Lustre: DEBUG MARKER: Force oleg138-client.virtnet to discover 192.168.201.138@tcp (in background) [ 1714.276470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.138@tcp [ 1716.069719] Lustre: DEBUG MARKER: Force oleg138-server to discover 192.168.201.38@tcp [ 1717.343611] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1719.690762] Lustre: DEBUG MARKER: Wait for 71878 [ 1720.682572] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 1721.630816] Lustre: DEBUG MARKER: Finished wait on 71878 [ 1721.726620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1721.823941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1722.212200] LNet: 72333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.218291] LNetError: 72333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.243404] LNet: Removed LNI 192.168.201.38@tcp [ 1723.999699] Key type .llcrypt unregistered [ 1724.005302] Key type ._llcrypt unregistered [ 1743.976492] Key type ._llcrypt registered [ 1743.980868] Key type .llcrypt registered [ 1745.534295] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:17:37 (1788941857) [ 1746.133162] Key type .llcrypt unregistered [ 1746.149692] Key type ._llcrypt unregistered [ 1753.132154] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1753.251334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1753.263783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1753.266720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1756.078236] Key type ._llcrypt registered [ 1756.080290] Key type .llcrypt registered [ 1756.316222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1756.324210] alg: No test for adler32 (adler32-zlib) [ 1757.366695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.444544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.472290] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1758.100868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1758.121331] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1768.707433] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:18:00 (1788941880) [ 1769.002327] LNet: 74409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.014252] LNetError: 74409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1769.022353] LNet: Removed LNI 10.1.2.3@tcp [ 1769.034762] LNet: Removed LNI 192.168.201.38@tcp [ 1769.956500] Key type .llcrypt unregistered [ 1769.958403] Key type ._llcrypt unregistered [ 1778.444126] Key type ._llcrypt registered [ 1778.453643] Key type .llcrypt registered [ 1778.633912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1778.650192] alg: No test for adler32 (adler32-zlib) [ 1779.631931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.740622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.758194] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1780.624046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1780.691476] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1782.076454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.38@tcp [ 1783.158233] LNetError: Unexpected error -22 connecting to 192.168.201.38@tcp at host 192.168.201.38:988 [ 1789.879949] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:18:21 (1788941901) [ 1789.974435] LNet: 75530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.988779] LNetError: 75530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1790.003069] LNet: Removed LNI 10.1.2.3@tcp [ 1791.017655] LNet: Removed LNI 192.168.201.38@tcp [ 1791.867781] Key type .llcrypt unregistered [ 1791.878353] Key type ._llcrypt unregistered [ 1801.643377] Key type ._llcrypt registered [ 1801.660487] Key type .llcrypt registered [ 1801.965464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1801.988805] alg: No test for adler32 (adler32-zlib) [ 1802.999377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.076675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1803.117426] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1803.713632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1803.731436] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 1804.540308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.38@tcp1 --nid 192.168.201.38@tcp2 [ 1809.390754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp1 192.168.201.38@tcp2 [ 1809.449674] LNetError: 76372:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.38@tcp1 to 192.168.201.38@tcp2 - Route table may be misconfigured [ 1809.497278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp2 192.168.201.38@tcp1 [ 1816.491847] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:18:48 (1788941928) [ 1816.961779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.001240] LNet: 76946:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.010451] LNetError: 76946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.015178] LNet: Removed LNI 192.168.201.38@tcp1 [ 1818.018052] LNetError: 76946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.040962] LNet: Removed LNI 192.168.201.38@tcp2 [ 1818.106920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.216043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.238891] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1819.142034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1819.191813] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1820.193407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 1820.230851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.38@tcp [ 1820.275889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp1 192.168.201.38@tcp1 [ 1820.333738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.38@tcp -d 192.168.201.38@tcp -e network_timeout [ 1820.371542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -e network_timeout [ 1820.405489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -e network_timeout [ 1820.441311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -e network_timeout [ 1820.508312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1820.552839] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 1820.590140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1820.723259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1820.864793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1821.280556] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 1827.320792] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:18:59 (1788941939) [ 1827.592775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.665871] LNet: 77806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.685951] LNetError: 77806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.694792] LNet: Removed LNI 192.168.201.38@tcp [ 1828.711671] LNet: Removed LNI 192.168.201.38@tcp1 [ 1828.780988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.000496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1829.438437] LNet: 77845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.327672] Key type .llcrypt unregistered [ 1830.331271] Key type ._llcrypt unregistered [ 1842.875301] Key type ._llcrypt registered [ 1842.878653] Key type .llcrypt registered [ 1844.527423] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:19:16 (1788941956) [ 1845.125138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.152593] alg: No test for adler32 (adler32-zlib) [ 1846.252573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.090098] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1850.422288] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:19:22 (1788941962) [ 1850.800103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.843532] LNet: 78941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.883351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.952942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.974928] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1852.071430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1852.118984] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1853.316639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1853.344043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp1 [ 1853.379553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp1 [ 1861.220991] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:19:33 (1788941973) [ 1862.678511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg138-server [ 1864.838965] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:19:36 (1788941976) [ 1867.099922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg138-server [ 1869.196277] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:19:40 (1788941980) [ 1871.180069] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg138-server [ 1873.186435] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:19:45 (1788941985) [ 1874.984113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg138-server [ 1877.043315] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:19:48 (1788941988) [ 1879.470560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg138-server [ 1881.617756] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:19:53 (1788941993) [ 1883.885262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg138-server [ 1886.041580] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:19:57 (1788941997) [ 1888.143811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg138-server [ 1890.054903] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:20:01 (1788942001) [ 1892.726646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg138-server [ 1895.215038] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:20:06 (1788942006) [ 1897.495738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg138-server [ 1900.261990] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:20:11 (1788942011) [ 1900.348172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.370889] LNet: 82815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.388401] LNetError: 82815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.396389] LNet: Removed LNI 192.168.201.38@tcp [ 1901.414445] LNet: Removed LNI 192.168.201.38@tcp1 [ 1901.493308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.580135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.610451] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1902.640416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1902.767356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1903.033327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.057557] LNet: 82971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.063335] LNetError: 82971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.077672] LNetError: 82971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1903.098269] LNet: Removed LNI 192.168.201.38@tcp [ 1903.146959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.243380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1903.315942] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1904.369368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1904.607680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.640273] LNetError: 83116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.714463] LNet: Removed LNI 192.168.201.38@tcp [ 1905.756873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.866299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.898469] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1906.672842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1906.757244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1906.992916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.016857] LNet: 83263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.034336] LNet: 83263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1907.047475] LNetError: 83263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.082472] LNet: Removed LNI 192.168.201.38@tcp [ 1907.115609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.161717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1907.183772] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1907.847098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1908.117677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.159954] LNet: Removed LNI 192.168.201.38@tcp [ 1909.184322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.247208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.287912] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1910.044917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1910.153866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1910.431332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.561319] LNet: Removed LNI 192.168.201.38@tcp [ 1911.600689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.710153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1911.747757] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1912.621790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1912.806161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.828294] LNet: 83702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1912.833246] LNet: 83702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1912.841901] LNetError: 83702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.848469] LNetError: 83702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1912.865103] LNet: Removed LNI 192.168.201.38@tcp [ 1912.888480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.951797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.969856] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1913.661315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1913.787579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1914.021487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.109736] LNet: Removed LNI 192.168.201.38@tcp [ 1915.170269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.266663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1915.303395] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1916.234757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1916.435398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.481847] LNet: Removed LNI 192.168.201.38@tcp [ 1917.510981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.609241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.637387] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1918.402657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1918.478346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1920.014000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.065581] LNet: Removed LNI 192.168.201.38@tcp [ 1921.103247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.213597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1921.242555] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1922.095904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1922.457585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.495823] LNet: 84296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.499966] LNet: 84296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1922.516752] LNetError: 84296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.525849] LNetError: 84296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1923.568318] LNet: Removed LNI 192.168.201.38@tcp [ 1923.597606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.694566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1923.725653] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1924.527779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1924.652527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1926.177431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.204885] LNet: Removed LNI 192.168.201.38@tcp [ 1927.294449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.358241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1927.418293] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1928.313645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 1928.671554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.714238] LNet: Removed LNI 192.168.201.38@tcp [ 1929.763887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.911838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.965912] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1931.214734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1931.301975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.329976] LNet: Removed LNI 192.168.201.38@tcp [ 1931.371799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.465560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1931.499111] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1932.601482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1932.714328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.761846] LNet: Removed LNI 192.168.201.38@tcp [ 1932.790191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.875611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1932.998546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.035577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.102274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1933.189373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.246579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.354955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1933.456223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1933.509831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.593278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1933.647816] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1934.573437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.603150] LNet: Removed LNI 192.168.201.38@tcp [ 1934.631327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.706845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1941.888949] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:20:53 (1788942053) [ 1942.032392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.073581] LNet: 85612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.079053] LNet: 85612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1942.117654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.233914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1942.267655] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/180] [ 1943.524334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.550695] LNetError: 85757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.556203] LNetError: 85757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1943.572176] LNet: Removed LNI 192.168.201.38@tcp231 [ 1943.602929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.741660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1943.789512] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/180] [ 1943.797027] LNet: 85765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1944.312416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1944.357922] LNet: Removed LNI 192.168.201.38@tcp231 [ 1944.383561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1944.444945] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/180] [ 1945.627529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.654790] LNet: Removed LNI 192.168.201.38@tcp231 [ 1945.706540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.778859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1945.808979] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/0] [ 1945.814789] LNet: 85918:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1946.326281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1946.365687] LNet: Removed LNI 192.168.201.38@tcp231 [ 1946.382623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1946.417896] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/0] [ 1947.588523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.648870] LNet: Removed LNI 192.168.201.38@tcp231 [ 1947.676643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.734380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1947.766985] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/60] [ 1947.780090] LNet: 86069:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1948.371886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1948.416526] LNet: Removed LNI 192.168.201.38@tcp231 [ 1948.441613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1948.465262] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/60] [ 1949.675485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.707877] LNet: Removed LNI 192.168.201.38@tcp231 [ 1949.747793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.852579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1949.873772] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/180] [ 1949.884723] LNet: 86220:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1950.303278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1950.336433] LNet: Removed LNI 192.168.201.38@tcp231 [ 1950.356951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1950.396352] LNet: Added LNI 192.168.201.38@tcp231 [8/256/0/180] [ 1958.851555] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:21:10 (1788942070) [ 1958.981613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.037956] LNet: Removed LNI 192.168.201.38@tcp231 [ 1959.066718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.190565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1959.213919] LNet: Added LNI 192.168.201.38@tcp232 [8/256/0/180] [ 1960.128765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1960.175165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1960.205697] LNet: Removed LNI 192.168.201.38@tcp232 [ 1960.238831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1960.282060] LNet: Added LNI 192.168.201.38@tcp232 [8/256/0/180] [ 1961.146325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1961.199900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1961.242959] LNet: Removed LNI 192.168.201.38@tcp232 [ 1961.277571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1961.322278] LNet: Added LNI 192.168.201.38@tcp232 [8/256/0/180] [ 1962.423853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1962.492258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1962.547992] LNet: Removed LNI 192.168.201.38@tcp232 [ 1962.575612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1962.610435] LNet: Added LNI 192.168.201.38@tcp232 [8/256/0/180] [ 1963.194539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1963.233874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1963.280080] LNet: Removed LNI 192.168.201.38@tcp232 [ 1963.318649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1970.105788] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:21:21 (1788942081) [ 1971.415570] Key type .llcrypt unregistered [ 1971.418125] Key type ._llcrypt unregistered [ 1978.430377] Key type ._llcrypt registered [ 1978.434514] Key type .llcrypt registered [ 1978.631043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1978.642331] alg: No test for adler32 (adler32-zlib) [ 1979.635364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1979.677206] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 1988.211801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 1997.050988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.130995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1997.184273] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 1999.680233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_interrupt -r 2 [ 1999.756704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_interrupt -r 2 [ 1999.839318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_interrupt -r 2 [ 1999.897382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_interrupt -r 2 [ 1999.949955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_interrupt -r 2 [ 2000.003711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_interrupt -r 2 [ 2000.047559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_interrupt -r 2 [ 2000.120373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_interrupt -r 2 [ 2000.188540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2000.207564] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 2000.213827] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 2000.514305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2000.569908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2000.614258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2000.647559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2000.682280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2000.723731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2000.773070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2000.819155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2000.862059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2000.878569] LNet: 88722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2000.905800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2000.939766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2000.971730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2001.011873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2001.064882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2001.110837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2001.677913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_dropped -r 2 [ 2001.712336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_dropped -r 2 [ 2001.753176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_dropped -r 2 [ 2001.799501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_dropped -r 2 [ 2001.856636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_dropped -r 2 [ 2001.903673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_dropped -r 2 [ 2001.948628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_dropped -r 2 [ 2001.983672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_dropped -r 2 [ 2002.009241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2002.030800] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 2002.262236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2002.285227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2002.303035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2002.331516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2002.360797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2002.389535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2002.434311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2002.539523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2002.599379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2002.624406] LNet: 88844:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2002.666794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2002.716106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2002.745275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2002.769674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2002.814319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2002.856300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2003.486293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_aborted -r 2 [ 2003.548333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_aborted -r 2 [ 2003.606971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_aborted -r 2 [ 2003.667340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_aborted -r 2 [ 2003.707751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_aborted -r 2 [ 2003.759305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_aborted -r 2 [ 2003.796622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_aborted -r 2 [ 2003.829989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_aborted -r 2 [ 2003.861705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2003.882772] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 2004.193279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2004.221822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2004.266204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2004.332832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2004.390785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2004.442120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2004.489514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2004.513931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2004.537117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2004.551453] LNet: 88967:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2004.562576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2004.581821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2004.607166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2004.632150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2004.660456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2004.695506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2005.356453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_no_route -r 2 [ 2005.387603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_no_route -r 2 [ 2005.422145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_no_route -r 2 [ 2005.454581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_no_route -r 2 [ 2005.494490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_no_route -r 2 [ 2005.538587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_no_route -r 2 [ 2005.580175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_no_route -r 2 [ 2005.625595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_no_route -r 2 [ 2005.669360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2005.963789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2006.002426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2006.053404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2006.083404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2006.108776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2006.135323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2006.163965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2006.183523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2006.207379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2006.247634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2006.297912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2006.340714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2006.391445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2006.441468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2006.494286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2007.132739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e local_timeout -r 2 [ 2007.202065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e local_timeout -r 2 [ 2007.263156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e local_timeout -r 2 [ 2007.331823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e local_timeout -r 2 [ 2007.403076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e local_timeout -r 2 [ 2007.490810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e local_timeout -r 2 [ 2007.549566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e local_timeout -r 2 [ 2007.620940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e local_timeout -r 2 [ 2007.662650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2007.691793] LNet: There was an unexpected network error while writing to 192.168.201.138: rc = -22 [ 2007.700284] LNet: Skipped 1 previous similar message [ 2008.095728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2008.142730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2008.181305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2008.201393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2008.257465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2008.324081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2008.392844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2008.435759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2008.505044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2008.537321] LNet: 89211:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2008.548853] LNet: 89211:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2008.571407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2008.644609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2008.694459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2008.752161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2008.808118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2008.863704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2009.442276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp -m GET -e remote_dropped -r 2 [ 2009.484358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.138@tcp1 -m GET -e remote_dropped -r 2 [ 2009.536431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp -m GET -e remote_dropped -r 2 [ 2009.580688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp -d 192.168.201.38@tcp1 -m GET -e remote_dropped -r 2 [ 2009.613590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp -m GET -e remote_dropped -r 2 [ 2009.644681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.138@tcp1 -m GET -e remote_dropped -r 2 [ 2009.680591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp -m GET -e remote_dropped -r 2 [ 2009.711713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.38@tcp1 -d 192.168.201.38@tcp1 -m GET -e remote_dropped -r 2 [ 2009.750884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.38@tcp 192.168.201.138@tcp [ 2009.769441] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.138@tcp [ 2010.082699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2010.111752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2010.133747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2010.158593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2010.199490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2010.229578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2010.271291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2010.303249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2010.351976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2010.405498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2010.449250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2010.507566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2010.549441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2010.597697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2010.654106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2017.489230] LNet: 89408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2017.507510] LNetError: 89408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.523535] LNet: Removed LNI 192.168.201.38@tcp [ 2017.556636] LNet: Removed LNI 192.168.201.38@tcp1 [ 2017.960599] Key type .llcrypt unregistered [ 2017.967496] Key type ._llcrypt unregistered [ 2039.325128] Key type ._llcrypt registered [ 2039.328399] Key type .llcrypt registered [ 2040.932975] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:22:32 (1788942152) [ 2041.790047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.841778] alg: No test for adler32 (adler32-zlib) [ 2042.766467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.822820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.857683] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2044.045861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2044.119180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.38@tcp [ 2044.158981] LNetError: 90385:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.38@tcp to recovery [ 2044.238400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.38@tcp [ 2044.327299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2050.910805] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:22:42 (1788942162) [ 2051.241617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2051.266909] LNet: 90974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.272241] LNetError: 90974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.297328] LNet: Removed LNI 192.168.201.38@tcp [ 2051.321333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.402245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2051.424415] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2052.151987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2052.171842] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 2053.182165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.38@tcp [ 2053.197518] LNetError: 91240:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.38@tcp to recovery [ 2053.410612] LNet: 1 local NIs in recovery (showing 1): 192.168.201.38@tcp [ 2053.415830] LNet: There was an unexpected network error while writing to 192.168.201.38: rc = -22 [ 2054.469593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.38@tcp [ 2062.941686] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:22:54 (1788942174) [ 2066.036415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg138-server [ 2068.654569] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:23:00 (1788942180) [ 2070.956450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg138-server [ 2073.271822] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:23:04 (1788942184) [ 2075.772495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg138-server [ 2077.930511] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:23:09 (1788942189) [ 2078.090648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.104608] LNet: 92862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.112302] LNetError: 92862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2078.120953] LNet: Removed LNI 192.168.201.38@tcp [ 2078.133577] LNet: Removed LNI 192.168.201.38@tcp1 [ 2078.174865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.221868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2078.254383] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2079.245619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2079.436541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2079.575330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2079.815387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2079.855278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2079.976125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2087.406899] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:23:19 (1788942199) [ 2088.034532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.060460] LNet: 93594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.069364] LNetError: 93594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.073810] LNetError: 93594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2088.094960] LNet: Removed LNI 192.168.201.38@tcp [ 2088.113948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.141821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.157343] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2096.017658] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:23:27 (1788942207) [ 2096.160457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.199763] LNet: 94290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.210104] LNetError: 94290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.249986] LNet: Removed LNI 192.168.201.38@tcp [ 2096.278872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.393702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.442370] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2104.136473] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:23:36 (1788942216) [ 2105.851979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2107.629784] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:23:39 (1788942219) [ 2108.493662] LNet: 95332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.509062] LNetError: 95332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.534740] LNet: Removed LNI 192.168.201.38@tcp [ 2109.573890] Key type .llcrypt unregistered [ 2109.576338] Key type ._llcrypt unregistered [ 2119.165627] Key type ._llcrypt registered [ 2119.170331] Key type .llcrypt registered [ 2119.687629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.709437] alg: No test for adler32 (adler32-zlib) [ 2120.810887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2120.881320] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2131.306388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 2137.844687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.138@tcp [ 2138.391841] LNet: 95748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.405684] LNetError: 95748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.431271] LNet: Removed LNI 192.168.201.38@tcp [ 2138.830235] Key type .llcrypt unregistered [ 2138.832882] Key type ._llcrypt unregistered [ 2151.504519] Key type ._llcrypt registered [ 2151.508493] Key type .llcrypt registered [ 2153.090918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2154.982526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2157.126799] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:24:29 (1788942269) [ 2158.912426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2160.530990] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:24:32 (1788942272) [ 2163.511508] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg138-server [ 2165.532345] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:24:37 (1788942277) [ 2167.450858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg138-server [ 2169.779629] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:24:41 (1788942281) [ 2171.674636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2171.719559] alg: No test for adler32 (adler32-zlib) [ 2172.877267] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 2173.125495] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2174.871142] Key type lgssc registered [ 2176.101898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.398314] Key type lgssc unregistered [ 2198.841212] LNet: 98921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.852349] LNetError: 98921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.888732] LNet: Removed LNI 192.168.201.38@tcp [ 2200.022878] Key type .llcrypt unregistered [ 2200.026763] Key type ._llcrypt unregistered [ 2217.312905] Key type ._llcrypt registered [ 2217.325909] Key type .llcrypt registered [ 2218.657959] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:25:30 (1788942330) [ 2219.235806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.274598] alg: No test for adler32 (adler32-zlib) [ 2220.206576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.271430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2220.305989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2220.342906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2220.374473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2227.683773] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:25:39 (1788942339) [ 2228.035392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.053883] LNet: 100333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2228.076606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.138643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.155395] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2229.088627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.111787] LNet: 100477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2229.117200] LNetError: 100477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.127929] LNet: Removed LNI 192.168.201.38@tcp [ 2229.153729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.228497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2229.258813] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2229.843570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.872224] LNetError: 100506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2229.898520] LNet: Removed LNI 192.168.201.38@tcp [ 2229.913506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.019695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2230.056167] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2237.869198] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:25:50 (1788942350) [ 2238.145961] LNet: 101096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.154441] LNet: 101096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2238.163903] LNetError: 101096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.184910] LNet: Removed LNI 192.168.201.38@tcp [ 2238.734816] Key type .llcrypt unregistered [ 2238.737506] Key type ._llcrypt unregistered [ 2245.208391] Key type ._llcrypt registered [ 2245.213549] Key type .llcrypt registered [ 2245.584746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2245.623486] alg: No test for adler32 (adler32-zlib) [ 2246.644075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.795584] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2246.798076] LNet: *** cfs_fail_loc=e003, val=100*** [ 2247.522734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2247.542931] LNet: *** cfs_fail_loc=e003, val=100*** [ 2247.545952] LNet: Skipped 2 previous similar messages [ 2247.551363] Ping reply received with mapping: [ 2247.553499] found metadata reply NI[192.168.201.38@tcp] [ 2247.556913] 26 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2247.560665] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2247.569439] 5a 5a 5a 5a 5a 5a 5a 5a [ 2247.814273] LNet: 101490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.821188] LNetError: 101490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.841799] LNet: Removed LNI 192.168.201.38@tcp [ 2248.646586] Key type .llcrypt unregistered [ 2248.649524] Key type ._llcrypt unregistered [ 2263.870272] Key type ._llcrypt registered [ 2263.872229] Key type .llcrypt registered [ 2265.648279] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:26:17 (1788942377) [ 2266.280443] Key type .llcrypt unregistered [ 2266.283702] Key type ._llcrypt unregistered [ 2274.685908] Key type ._llcrypt registered [ 2274.687605] Key type .llcrypt registered [ 2275.075387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2275.120333] alg: No test for adler32 (adler32-zlib) [ 2276.520821] LNetError: 102527:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2277.199462] Key type .llcrypt unregistered [ 2277.201366] Key type ._llcrypt unregistered [ 2283.588786] Key type ._llcrypt registered [ 2283.600860] Key type .llcrypt registered [ 2285.543319] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:26:36 (1788942396) [ 2286.192630] Key type .llcrypt unregistered [ 2286.206658] Key type ._llcrypt unregistered [ 2293.217661] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2293.220661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2295.950133] Key type ._llcrypt registered [ 2295.971688] Key type .llcrypt registered [ 2296.135961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2296.144675] alg: No test for adler32 (adler32-zlib) [ 2297.094901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.193596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2308.285321] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:26:59 (1788942419) [ 2308.739959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.766170] LNet: 104302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.793571] LNetError: 104302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2308.810959] LNet: Removed LNI 10.1.2.3@tcp [ 2308.831445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.972761] Key type .llcrypt unregistered [ 2309.996328] Key type ._llcrypt unregistered [ 2319.442165] Key type ._llcrypt registered [ 2319.455330] Key type .llcrypt registered [ 2319.710710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.740867] alg: No test for adler32 (adler32-zlib) [ 2320.626682] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2320.724675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2320.984857] LNet: 104588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.988920] LNetError: 104588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.000610] LNet: Removed LNI 192.168.201.38@tcp [ 2321.607596] Key type .llcrypt unregistered [ 2321.611665] Key type ._llcrypt unregistered [ 2338.663140] Key type ._llcrypt registered [ 2338.665399] Key type .llcrypt registered [ 2340.265420] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:27:31 (1788942451) [ 2341.665580] Key type .llcrypt unregistered [ 2341.668679] Key type ._llcrypt unregistered [ 2349.640066] Key type ._llcrypt registered [ 2349.642153] Key type .llcrypt registered [ 2349.910835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2349.962482] alg: No test for adler32 (adler32-zlib) [ 2358.150548] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:27:49 (1788942469) [ 2358.408743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.433524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.482385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.509784] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2359.379359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2359.436929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2359.487374] LNet: Removed LNI 192.168.201.38@tcp [ 2359.866948] LNet: 106329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2360.387618] Key type .llcrypt unregistered [ 2360.389739] Key type ._llcrypt unregistered [ 2374.765674] Key type ._llcrypt registered [ 2374.767613] Key type .llcrypt registered [ 2376.318667] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:28:08 (1788942488) [ 2376.929684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.941222] alg: No test for adler32 (adler32-zlib) [ 2377.985381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.050173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.083911] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2378.906841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2378.955474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.38@tcp [ 2378.990087] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2378.993527] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2379.002856] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2378, 0, 0 [ 2379.014405] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2379.023273] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2379.030756] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2379.041172] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2378, 0, 0 [ 2379.056942] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2379.061686] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2379.075660] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2379.082951] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2378, 0, 2428 [ 2379.088403] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2379.090590] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2379.095056] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2379.099948] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2378, 0, 2428 [ 2379.102293] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2386.659742] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:28:18 (1788942498) [ 2387.139514] LNet: 107866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.144250] LNetError: 107866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.163577] LNet: Removed LNI 192.168.201.38@tcp [ 2387.752558] Key type .llcrypt unregistered [ 2387.758657] Key type ._llcrypt unregistered [ 2395.437743] Key type ._llcrypt registered [ 2395.439606] Key type .llcrypt registered [ 2395.736790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2395.765820] alg: No test for adler32 (adler32-zlib) [ 2396.659524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2396.694086] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2406.470641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 2414.911965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.015389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2415.044548] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 2415.978774] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2416.179581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2416.209400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2416.218379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2418.583805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2418.604411] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2428.835217] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:29:01 (1788942541) [ 2428.985454] LNet: 109366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.006372] LNetError: 109366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.015690] LNet: Removed LNI 192.168.201.38@tcp [ 2430.047801] LNet: Removed LNI 192.168.201.38@tcp1 [ 2430.052990] LNetError: 109366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2430.057654] LNetError: 109366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2430.076160] LNet: Removed LNI 10.1.2.3@tcp99 [ 2430.862454] Key type .llcrypt unregistered [ 2430.868667] Key type ._llcrypt unregistered [ 2437.939224] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2437.943681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2440.779111] Key type ._llcrypt registered [ 2440.781341] Key type .llcrypt registered [ 2441.060649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2441.074711] alg: No test for adler32 (adler32-zlib) [ 2442.051502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.107764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.130388] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2442.966727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2442.982192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2445.070914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2445.147386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2445.232363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2445.246837] LNetError: 110112:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2445.266839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2445.316142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2445.745898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2445.805569] LNet: 110132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.825370] LNetError: 110132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2445.830874] LNet: Removed LNI 10.1.2.3@tcp [ 2446.901344] LNet: Removed LNI 192.168.201.38@tcp [ 2447.639427] Key type .llcrypt unregistered [ 2447.643772] Key type ._llcrypt unregistered [ 2460.795807] Key type ._llcrypt registered [ 2460.797714] Key type .llcrypt registered [ 2462.268523] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 04:29:34 (1788942574) [ 2462.768219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.800584] alg: No test for adler32 (adler32-zlib) [ 2463.784934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.852680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2463.881628] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2464.785965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2464.888476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg138-client@tcp [ 2472.732925] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 04:29:44 (1788942584) [ 2472.997816] LNet: 111662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2473.002482] LNetError: 111662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.035161] LNet: Removed LNI 192.168.201.38@tcp [ 2474.783576] Key type .llcrypt unregistered [ 2474.790185] Key type ._llcrypt unregistered [ 2482.987748] Key type ._llcrypt registered [ 2482.993915] Key type .llcrypt registered [ 2483.380826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.447564] alg: No test for adler32 (adler32-zlib) [ 2484.477349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2484.516588] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2484.559264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2491.739309] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 04:30:03 (1788942603) [ 2493.309195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2495.303697] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 04:30:07 (1788942607) [ 2497.068140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2499.281796] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 04:30:11 (1788942611) [ 2499.863838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2499.884477] LNet: 113091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.897635] LNetError: 113091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.917042] LNet: Removed LNI 192.168.201.38@tcp [ 2499.932700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.014452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.054769] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2500.816842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2500.889957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2507.283201] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 04:30:19 (1788942619) [ 2507.753155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2507.775207] LNet: 113802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2507.785268] LNetError: 113802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2507.798249] LNet: Removed LNI 192.168.201.38@tcp [ 2507.812888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2507.899914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2507.941379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2508.648720] Key type .llcrypt unregistered [ 2508.650733] Key type ._llcrypt unregistered [ 2521.527952] Key type ._llcrypt registered [ 2521.531492] Key type .llcrypt registered [ 2522.959039] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 04:30:34 (1788942634) [ 2523.320456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2523.336994] alg: No test for adler32 (adler32-zlib) [ 2524.276427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.360810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2524.415790] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2525.307106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2525.401410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2532.429537] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 04:30:44 (1788942644) [ 2532.574863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2532.632714] LNet: 115348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.639489] LNetError: 115348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.681105] LNet: Removed LNI 192.168.201.38@tcp [ 2532.741893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2532.788696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2532.819261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2540.509804] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 04:30:52 (1788942652) [ 2540.813243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2540.860068] LNet: 115934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2540.895159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2540.964793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2541.516452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2541.558459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2541.642539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2542.240058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2542.292958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2542.413604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2542.446786] LNet: 115968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2542.478901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2542.579808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2543.066054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2543.120299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2543.214438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2543.749349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2543.798103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2551.180422] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 04:31:03 (1788942663) [ 2551.470184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2551.487823] LNet: 116568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2551.504296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.551325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2551.591271] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2552.244708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2552.265668] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2552.974809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2553.081925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2553.320127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2553.338978] LNetError: 116847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.344403] LNet: Removed LNI 192.168.201.38@tcp [ 2553.353238] LNet: Removed LNI 192.168.201.38@tcp2 [ 2553.374168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2553.480854] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2553.496402] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2553.503546] LNet: 116853:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2554.109377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2554.284977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2555.359700] LNet: Removed LNI 192.168.201.38@tcp [ 2555.362842] LNetError: 116891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2555.372610] LNetError: 116891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2555.415219] LNet: Removed LNI 192.168.201.38@tcp2 [ 2555.451067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2555.537525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2555.579991] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2556.521796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2556.542790] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2557.409667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2557.480778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2557.573942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2557.886806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2557.903607] LNet: 117175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.914853] LNet: 117175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2557.923771] LNetError: 117175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.936515] LNet: Removed LNI 192.168.201.38@tcp [ 2558.958936] LNet: Removed LNI 192.168.201.38@tcp2 [ 2558.977244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2559.022713] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2559.027442] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2559.031814] LNet: 117181:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2559.568427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2559.886824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2559.929948] LNet: Removed LNI 192.168.201.38@tcp [ 2561.004190] LNet: Removed LNI 192.168.201.38@tcp2 [ 2561.087112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2561.211495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2561.251682] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2562.127810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2562.147927] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2563.303614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2563.402381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2563.710878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2563.735228] LNetError: 117498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.740091] LNetError: 117498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2564.767163] LNet: Removed LNI 192.168.201.38@tcp [ 2564.788497] LNet: Removed LNI 192.168.201.38@tcp2 [ 2564.874726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2564.949712] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2564.963954] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2564.968775] LNet: 117504:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2565.547975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2565.773571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2566.815358] LNet: Removed LNI 192.168.201.38@tcp [ 2566.830671] LNet: Removed LNI 192.168.201.38@tcp2 [ 2566.860598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2566.951569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2566.978984] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2567.806650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2567.823845] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2568.831970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2568.916765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2569.061055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2569.359246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2569.375511] LNet: 117826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.379526] LNet: 117826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2570.402580] LNet: Removed LNI 192.168.201.38@tcp [ 2571.433581] LNet: Removed LNI 192.168.201.38@tcp2 [ 2571.458742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2571.529391] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2571.540445] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2571.552411] LNet: 117832:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2572.014733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2572.235285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2572.252792] LNetError: 117869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2572.258197] LNetError: 117869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2572.265754] LNet: Removed LNI 192.168.201.38@tcp [ 2573.289885] LNet: Removed LNI 192.168.201.38@tcp2 [ 2573.381162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2573.485074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2573.532401] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2574.089076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2574.110193] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2574.871550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2574.917544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2574.972514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2575.309674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2576.351155] LNet: Removed LNI 192.168.201.38@tcp [ 2576.361176] LNet: Removed LNI 192.168.201.38@tcp2 [ 2576.381239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2576.459535] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2576.464659] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2576.469141] LNet: 118173:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2577.114313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2577.415602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2578.463434] LNet: Removed LNI 192.168.201.38@tcp [ 2578.482614] LNet: Removed LNI 192.168.201.38@tcp2 [ 2578.536948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2578.632452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2578.676512] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2579.490995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2579.525100] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2580.485111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2580.508112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2580.546593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2580.591792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2580.952472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2581.984089] LNet: Removed LNI 192.168.201.38@tcp [ 2582.006961] LNet: Removed LNI 192.168.201.38@tcp2 [ 2582.054499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2582.150762] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2582.170404] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2582.772863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2589.990870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2591.809879] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 04:31:43 (1788942703) [ 2592.217275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2592.236645] LNet: 119291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.240284] LNet: 119291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2592.244836] LNetError: 119291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2592.250960] LNetError: 119291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2592.256243] LNet: Removed LNI 192.168.201.38@tcp [ 2593.334040] LNet: Removed LNI 192.168.201.38@tcp2 [ 2593.382182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2593.719577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2593.749347] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2594.569407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2594.590950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2595.638651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2595.684469] LNet: Removed LNI 10.1.2.3@tcp [ 2595.701480] LNet: Removed LNI 192.168.201.38@tcp [ 2595.756064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2595.870638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2595.917227] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2595.941860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2596.590740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2596.622668] LNet: Removed LNI 192.168.201.38@tcp [ 2596.641770] LNet: Removed LNI 10.1.2.3@tcp [ 2603.515552] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 04:31:55 (1788942715) [ 2603.823702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2603.888307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.934625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2603.959173] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2604.874218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2604.929058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2605.021979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.078956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.186781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.250666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.316639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.362417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.453126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.496098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.551801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.625142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2605.843134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2612.787293] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 04:32:04 (1788942724) [ 2613.126320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2614.263163] LNet: Removed LNI 192.168.201.38@tcp [ 2614.310532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2614.420913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2614.494768] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2615.485473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2615.528787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2615.607981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.668709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.729643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.794577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.839767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.880616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.931535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2615.991879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.026461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.062457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.184774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2616.272833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.325759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.363455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.414918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.484148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.554935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.610503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.677791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.724121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2616.778753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.38@tcp [ 2623.224643] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 04:32:15 (1788942735) [ 2623.465694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2624.557294] LNet: Removed LNI 192.168.201.38@tcp [ 2624.603638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2624.743235] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2625.045124] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2625.052618] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2625.071268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2627.373294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2627.413118] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2628.061341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2628.077302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2630.366549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2630.415603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2630.540939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.38@tcp -m PUT -e local_timeout -r 1 [ 2630.619783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2630.679742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2630.688262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2631.026658] LNetError: 121726:0:(lib-move.c:4794:lnet_parse()) 192.168.201.38@tcp, src 192.168.201.38@tcp: Dropping PUT (error -108 looking up sender) [ 2632.050645] LNet: 122186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.070500] LNet: 122186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2632.087758] LNetError: 122186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2632.108796] LNetError: 122186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2632.131910] LNet: Removed LNI 10.1.2.3@tcp [ 2633.196181] LNet: Removed LNI 192.168.201.38@tcp [ 2633.768322] Key type .llcrypt unregistered [ 2633.770604] Key type ._llcrypt unregistered [ 2648.103520] Key type ._llcrypt registered [ 2648.105356] Key type .llcrypt registered [ 2649.539363] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 04:32:41 (1788942761) [ 2649.884877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2649.901739] alg: No test for adler32 (adler32-zlib) [ 2650.920900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2651.349841] LNet: 123036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2652.050587] Key type .llcrypt unregistered [ 2652.053209] Key type ._llcrypt unregistered [ 2658.391792] Key type ._llcrypt registered [ 2658.395436] Key type .llcrypt registered [ 2658.601306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2658.620618] alg: No test for adler32 (adler32-zlib) [ 2659.616516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2659.652942] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2668.096745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 2668.303983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2668.990549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.138@tcp [ 2669.318422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2669.346763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2669.381803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2669.409400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2669.446601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2669.489648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2669.525296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2669.565349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2669.605443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2669.633265] LNet: 123485:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2669.654734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2669.696697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2669.738832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2669.776869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2669.848526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2669.894050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2670.707227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2670.944297] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.138@tcp [ 2671.002944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2671.033252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2671.059979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2671.090504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2671.117684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2671.150668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2671.179295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2671.207595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2671.248325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2671.274906] LNet: 123604:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2671.299439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2671.334528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2671.379320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2671.406979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2671.443544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2671.479342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2672.241649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.138@tcp [ 2672.500459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2672.535887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2672.566572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2672.596672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2672.623407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2672.658569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2672.693464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2672.732237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2672.772325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2672.792547] LNet: 123723:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2672.821324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2672.855728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2672.892448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2672.929495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2672.964982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2672.998223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2673.659911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2673.918879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2673.950210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2673.979331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2674.019927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2674.049868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2674.084836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2674.134749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2674.172831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2674.197798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2674.227594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2674.262647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2674.293514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2674.334489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2674.388325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2674.441160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2674.847661] LNet: 123888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.853562] LNetError: 123888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.869938] LNet: Removed LNI 192.168.201.38@tcp [ 2675.386641] Key type .llcrypt unregistered [ 2675.389892] Key type ._llcrypt unregistered [ 2691.316869] Key type ._llcrypt registered [ 2691.318558] Key type .llcrypt registered [ 2692.695640] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 04:33:24 (1788942804) [ 2692.998407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.012410] alg: No test for adler32 (adler32-zlib) [ 2693.967405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2694.425679] LNet: 124772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.273476] Key type .llcrypt unregistered [ 2695.283696] Key type ._llcrypt unregistered [ 2703.670226] Key type ._llcrypt registered [ 2703.672762] Key type .llcrypt registered [ 2703.960548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2703.975738] alg: No test for adler32 (adler32-zlib) [ 2705.045816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2705.099410] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2712.843854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.138@tcp [ 2722.723545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2722.797723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2722.850561] LNet: Added LNI 192.168.201.38@tcp1 [8/256/0/180] [ 2724.005068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2724.595886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.138@tcp [ 2724.881185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2724.924412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2724.993375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2725.048510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2725.091937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2725.134467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2725.164684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2725.195921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2725.231738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2725.248994] LNet: 125423:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2725.270697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2725.300912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2725.338560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2725.366729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2725.399669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2725.429693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2726.054973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2726.301120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2726.336465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2726.392325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2726.422168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2726.460546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2726.503166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2726.536174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2726.569990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2726.611462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2726.623171] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.138@tcp, 192.168.201.138@tcp1 [ 2726.644440] LNet: 125543:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2726.683574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2726.761338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2726.802251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2726.832849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2726.866526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2726.905576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2727.536986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.138@tcp [ 2727.765868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2727.792351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2727.823487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2727.844613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2727.872644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2727.906242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2727.940435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2727.983367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2728.024094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2728.045884] LNet: 125662:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2728.067911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2728.106969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2728.157322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2728.191309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2728.233470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2728.279990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2728.990835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2729.221742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2729.242602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2729.277644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2729.301837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2729.318593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2729.342658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2729.376932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2729.419988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2729.450721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2729.473527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2729.496306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2729.526712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2729.580669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2729.659813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2729.724801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2734.464608] LNet: 125862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.470837] LNetError: 125862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.476269] LNet: Removed LNI 192.168.201.38@tcp [ 2734.493819] LNet: Removed LNI 192.168.201.38@tcp1 [ 2734.915424] Key type .llcrypt unregistered [ 2734.919472] Key type ._llcrypt unregistered [ 2752.796583] Key type ._llcrypt registered [ 2752.798982] Key type .llcrypt registered [ 2754.129278] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 04:34:26 (1788942866) [ 2754.678564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2754.715751] alg: No test for adler32 (adler32-zlib) [ 2755.614698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2755.674430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2755.693797] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2756.494625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2756.558819] LNet: Added LNI 192.168.201.38@tcp2 [8/256/0/180] [ 2757.492455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.38@tcp [ 2757.541725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2758.559157] LNet: Removed LNI 192.168.201.38@tcp [ 2765.589361] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 04:34:37 (1788942877) [ 2767.280908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2769.252230] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 04:34:41 (1788942881) [ 2777.088615] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 04:34:48 (1788942888) [ 2784.697486] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 04:34:56 (1788942896) [ 2792.259211] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 04:35:04 (1788942904) [ 2800.028140] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 04:35:12 (1788942912) [ 2810.542208] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 04:35:21 (1788942921) [ 2821.526407] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 04:35:33 (1788942933) [ 2829.429266] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 04:35:41 (1788942941) [ 2836.057991] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 04:35:48 (1788942948) [ 2841.955531] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 04:35:54 (1788942954) [ 2848.967532] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 04:36:00 (1788942960) [ 2856.893856] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 04:36:08 (1788942968) [ 2864.167532] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 04:36:16 (1788942976) [ 2871.077868] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 04:36:23 (1788942983) [ 2879.611219] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 04:36:31 (1788942991) [ 2887.027499] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 04:36:38 (1788942998) [ 2896.292624] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 04:36:47 (1788943007) [ 2904.515560] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 04:36:56 (1788943016) [ 2904.648033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2904.691912] LNet: 137551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2904.707922] LNetError: 137551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2904.743952] LNet: Removed LNI 192.168.201.38@tcp2 [ 2904.813239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2904.951993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2904.975474] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2906.677514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2906.709325] LNet: 137698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2906.716812] LNetError: 137698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2906.731564] LNet: Removed LNI 192.168.201.38@tcp [ 2906.752975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2906.821431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2906.849690] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 2915.826292] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 04:37:07 (1788943027) [ 2926.327724] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 04:37:17 (1788943037) [ 2938.256351] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 04:37:28 (1788943048) [ 2947.326028] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 04:37:39 (1788943059) [ 2954.716503] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 04:37:46 (1788943066) [ 2962.762586] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 04:37:54 (1788943074) [ 2972.076044] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 04:38:03 (1788943083) [ 2981.135885] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 04:38:13 (1788943093) [ 2988.371359] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 04:38:20 (1788943100) [ 2996.516235] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 04:38:28 (1788943108) [ 3004.076708] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 04:38:35 (1788943115) [ 3011.510466] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 04:38:43 (1788943123) [ 3018.246615] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2756 sec ========== 04:38:50 (1788943130) [ 3018.844746] LNet: 145168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3018.850514] LNetError: 145168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3018.864560] LNet: Removed LNI 192.168.201.38@tcp [ 3019.515616] Key type .llcrypt unregistered [ 3019.518611] Key type ._llcrypt unregistered [ 3032.375404] Key type ._llcrypt registered [ 3032.378427] Key type .llcrypt registered [ 3032.991836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3033.008214] alg: No test for adler32 (adler32-zlib) [ 3034.155825] Lustre: Lustre: Build Version: 2.17.58_2_g930488d [ 3034.498724] LNet: Added LNI 192.168.201.38@tcp [8/256/0/180] [ 3036.183227] Key type lgssc registered [ 3037.204780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.553083] Lustre: Mounted lustre-client - version 2.17.58_2_g930488d [ 3109.540312] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3120.655554] Lustre: Unmounted lustre-client [ 3152.363449] Key type lgssc unregistered [ 3152.594086] LNet: 148117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.599939] LNetError: 148117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.624179] LNet: Removed LNI 192.168.201.38@tcp [ 3153.610238] Key type .llcrypt unregistered [ 3153.619564] Key type ._llcrypt unregistered