[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 476865308 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002303] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006648] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009125] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.019735] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023388] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027096] *** VALIDATE tmpfs *** [ 0.028382] *** VALIDATE proc *** [ 0.029219] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.031308] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034024] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.038135] debug: unmapping init [mem 0xffffffff8bc59000-0xffffffff8bc60fff] [ 0.040931] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041806] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042023] ... version: 2 [ 0.043016] ... bit width: 48 [ 0.044015] ... generic registers: 4 [ 0.045014] ... value mask: 0000ffffffffffff [ 0.046013] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048010] ... event mask: 000000070000000f [ 0.049368] rcu: Hierarchical SRCU implementation. [ 0.051521] smp: Bringing up secondary CPUs ... [ 0.052910] x86: Booting SMP configuration: [ 0.053030] .... node #0, CPUs: #1 #2 #3 [ 0.058013] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136137] node 0 deferred pages initialised in 73ms [ 0.141112] devtmpfs: initialized [ 0.142575] x86/mm: Memory block size: 128MB [ 0.146608] gcov: version magic: 0x41383552 [ 0.149386] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155575] pinctrl core: initialized pinctrl subsystem [ 0.157373] [ 0.157943] ************************************************************* [ 0.160022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162052] ** ** [ 0.164021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.167014] ** ** [ 0.169012] ** This means that this kernel is built to expose internal ** [ 0.171012] ** IOMMU data structures, which may compromise security on ** [ 0.174013] ** your system. ** [ 0.176008] ** ** [ 0.178015] ** If you see this message and you are not debugging the ** [ 0.181012] ** kernel, report this immediately to your vendor! ** [ 0.183011] ** ** [ 0.185012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.188010] ************************************************************* [ 0.190776] NET: Registered protocol family 16 [ 0.192557] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.195177] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201033] cpuidle: using governor menu [ 0.203012] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204726] PCI: Using configuration type 1 for base access [ 0.207159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.224072] cryptd: max_cpu_qlen set to 1000 [ 0.228326] ACPI: Added _OSI(Module Device) [ 0.230014] ACPI: Added _OSI(Processor Device) [ 0.231020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.233023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.241927] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.249372] ACPI: Interpreter enabled [ 0.251059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.252012] ACPI: Using IOAPIC for interrupt routing [ 0.254172] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.258289] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273892] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.277062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.280026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.284103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291157] acpiphp: Slot [2] registered [ 0.294198] acpiphp: Slot [5] registered [ 0.296391] acpiphp: Slot [6] registered [ 0.297000] acpiphp: Slot [3] registered [ 0.298015] acpiphp: Slot [4] registered [ 0.300239] acpiphp: Slot [7] registered [ 0.302182] acpiphp: Slot [8] registered [ 0.304142] acpiphp: Slot [9] registered [ 0.306124] acpiphp: Slot [10] registered [ 0.307088] acpiphp: Slot [11] registered [ 0.308077] acpiphp: Slot [12] registered [ 0.310087] acpiphp: Slot [13] registered [ 0.311083] acpiphp: Slot [14] registered [ 0.312104] acpiphp: Slot [15] registered [ 0.314091] acpiphp: Slot [16] registered [ 0.315058] acpiphp: Slot [17] registered [ 0.316059] acpiphp: Slot [18] registered [ 0.317114] acpiphp: Slot [19] registered [ 0.318215] acpiphp: Slot [20] registered [ 0.320111] acpiphp: Slot [21] registered [ 0.321103] acpiphp: Slot [22] registered [ 0.322107] acpiphp: Slot [23] registered [ 0.323060] acpiphp: Slot [24] registered [ 0.325101] acpiphp: Slot [25] registered [ 0.326090] acpiphp: Slot [26] registered [ 0.328102] acpiphp: Slot [27] registered [ 0.329037] acpiphp: Slot [28] registered [ 0.330097] acpiphp: Slot [29] registered [ 0.331075] acpiphp: Slot [30] registered [ 0.333115] acpiphp: Slot [31] registered [ 0.334067] PCI host bridge to bus 0000:00 [ 0.335018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.341032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.343022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.347041] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.350038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.352216] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.355318] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360543] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.372023] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.377048] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.380029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.383016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.387022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.390886] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.393794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400777] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.405016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.417023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.425815] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440661] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.453019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.458015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.479018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.493546] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.500023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.516027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.543026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.563372] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.568829] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.572880] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.580402] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.582272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.588056] iommu: Default domain type: Passthrough [ 0.589659] SCSI subsystem initialized [ 0.592281] ACPI: bus type USB registered [ 0.594205] usbcore: registered new interface driver usbfs [ 0.597131] usbcore: registered new interface driver hub [ 0.599126] usbcore: registered new device driver usb [ 0.600194] pps_core: LinuxPPS API ver. 1 registered [ 0.602017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.608166] PTP clock support registered [ 0.612101] EDAC MC: Ver: 3.0.0 [ 0.615237] PCI: Using ACPI for IRQ routing [ 0.618823] NetLabel: Initializing [ 0.619015] NetLabel: domain hash size = 128 [ 0.622014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.626094] NetLabel: unlabeled traffic allowed by default [ 0.628284] vgaarb: loaded [ 0.629516] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.631019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.640732] clocksource: Switched to clocksource kvm-clock [ 0.777599] VFS: Disk quotas dquot_6.6.0 [ 0.779100] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782105] *** VALIDATE ramfs *** [ 0.783390] *** VALIDATE hugetlbfs *** [ 0.785239] pnp: PnP ACPI init [ 0.787774] pnp: PnP ACPI: found 6 devices [ 0.805805] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.809495] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812178] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814723] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817466] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.820704] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824196] NET: Registered protocol family 2 [ 0.826913] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832334] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.838156] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.844217] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.848292] TCP: Hash tables configured (established 65536 bind 65536) [ 0.851792] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.856247] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860180] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.864243] NET: Registered protocol family 1 [ 0.867825] RPC: Registered named UNIX socket transport module. [ 0.870780] RPC: Registered udp transport module. [ 0.872781] RPC: Registered tcp transport module. [ 0.874357] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.876622] NET: Registered protocol family 44 [ 0.878295] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.880635] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.882961] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.885275] PCI: CLS 0 bytes, default 64 [ 0.887389] Unpacking initramfs... [ 2.512588] debug: unmapping init [mem 0xffff9aae3cc64000-0xffff9aae3ffcffff] [ 2.518597] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.524079] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 2.529978] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.270483] Initialise system trusted keyrings [ 3.272320] Key type blacklist registered [ 3.274563] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.287697] zbud: loaded [ 3.292472] *** VALIDATE nfs *** [ 3.293602] *** VALIDATE nfs4 *** [ 3.295953] pstore: using deflate compression [ 3.303110] Platform Keyring initialized [ 3.441177] NET: Registered protocol family 38 [ 3.442925] Key type asymmetric registered [ 3.445104] Asymmetric key parser 'x509' registered [ 3.447689] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.450784] io scheduler mq-deadline registered [ 3.452436] io scheduler kyber registered [ 3.453878] io scheduler bfq registered [ 3.455910] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.459479] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.465770] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.474850] ACPI: Power Button [PWRF] [ 3.485899] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.495383] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.507726] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.544139] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.578833] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.584427] Non-volatile memory driver v1.3 [ 3.586304] Linux agpgart interface v0.103 [ 3.623805] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.627669] vda: detected capacity change from 0 to 76787712 [ 3.651689] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.655700] vdb: detected capacity change from 0 to 1073741824 [ 3.668875] libphy: Fixed MDIO Bus: probed [ 3.676809] usbcore: registered new interface driver usbserial_generic [ 3.680452] usbserial: USB Serial support registered for generic [ 3.684470] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.692021] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.694073] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.700460] mousedev: PS/2 mouse device common for all mice [ 3.708277] rtc_cmos 00:05: RTC can wake from S4 [ 3.710321] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.712925] rtc_cmos 00:05: registered as rtc0 [ 3.722749] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.727822] intel_pstate: CPU model not supported [ 3.733469] hid: raw HID events driver (C) Jiri Kosina [ 3.738199] usbcore: registered new interface driver usbhid [ 3.743944] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.747081] usbhid: USB HID core driver [ 3.747321] drop_monitor: Initializing network drop monitor service [ 3.747479] Initializing XFRM netlink socket [ 3.748147] NET: Registered protocol family 10 [ 3.769584] Segment Routing with IPv6 [ 3.771355] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.772071] NET: Registered protocol family 17 [ 3.772464] mpls_gso: MPLS GSO support [ 3.791721] RAS: Correctable Errors collector initialized. [ 3.794020] AVX version of gcm_enc/dec engaged. [ 3.797170] AES CTR mode by8 optimization enabled [ 3.901684] sched_clock: Marking stable (3901475417, 0)->(4850148337, -948672920) [ 3.912222] registered taskstats version 1 [ 3.914682] Loading compiled-in X.509 certificates [ 3.917142] zswap: loaded using pool lzo/zbud [ 3.955501] Key type big_key registered [ 3.970544] Key type encrypted registered [ 3.972142] ima: No TPM chip found, activating TPM-bypass! [ 3.974954] ima: Allocated hash algorithm: sha1 [ 3.977142] ima: No architecture policies found [ 3.979126] evm: Initialising EVM extended attributes: [ 3.981463] evm: security.selinux [ 3.982920] evm: security.ima [ 3.984275] evm: security.capability [ 3.985941] evm: HMAC attrs: 0x1 [ 3.988831] rtc_cmos 00:05: setting system clock to 2026-09-07 20:30:57 UTC (1788813057) [ 3.997146] debug: unmapping init [mem 0xffffffff8cc03000-0xffffffff8cdfffff] [ 4.001296] debug: unmapping init [mem 0xffffffff8b982000-0xffffffff8bc58fff] [ 4.010412] Write protecting the kernel read-only data: 28672k [ 4.019941] debug: unmapping init [mem 0xffffffff8a003000-0xffffffff8a1fffff] [ 4.024433] debug: unmapping init [mem 0xffffffff8a914000-0xffffffff8a9fffff] [ 4.070672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.081468] systemd[1]: Detected virtualization kvm. [ 4.083190] systemd[1]: Detected architecture x86-64. [ 4.085473] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.123719] systemd[1]: No hostname configured. [ 4.126446] systemd[1]: Set hostname to . [ 4.129498] random: systemd: uninitialized urandom read (16 bytes read) [ 4.134113] systemd[1]: Initializing machine ID from random generator. [ 4.252220] random: ln: uninitialized urandom read (6 bytes read) [ 4.482207] random: systemd: uninitialized urandom read (16 bytes read) [ 4.486689] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.496900] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.506724] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.600252] device-mapper: uevent: version 1.0.3 [ 5.602796] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.932858] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ 7.055312] random: fast init done [ 7.116554] scsi host0: ata_piix [ 7.149258] scsi host1: ata_piix [ 7.154170] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.163726] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.501769] random: crng init done [ 12.504601] random: 7 urandom warning(s) missed due to ratelimiting [ 13.478850] dracut-initqueue[580]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.895546] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.418811] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.465389] SELinux: Disabled at runtime. [ 20.618447] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.629909] systemd[1]: Detected virtualization kvm. [ 20.634723] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.290176] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.298474] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.311048] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.319342] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.326984] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.340117] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.359171] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ 22.979497] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.903592] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.976219] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.018595] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.888325] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.995799] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 31.316601] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 32.155769] NFS: Registering the id_resolver key type [ 32.157823] Key type id_resolver registered [ *[0;1[ 32.165188] Key type id_legacy registered ;31m**] A start job is running for Configur…only root support (10s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ 35.072356] hrtimer: interrupt took 2919398 ns [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg425-client login: [ 120.897019] libcfs: loading out-of-tree module taints kernel. [ 120.947494] Key type ._llcrypt registered [ 120.949531] Key type .llcrypt registered [ 121.315686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 121.328833] alg: No test for adler32 (adler32-zlib) [ 122.832951] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 123.864790] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 125.663347] Key type lgssc registered [ 127.839946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 282.442814] Lustre: Mounted lustre-client - version 2.17.58_40_gea5fd3f [ 288.192531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.823918] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing check_logdir /tmp/testlogs/ [ 308.193721] Lustre: lustre-OST0000-osc-ffff9aae95778000: disconnect after 23s idle [ 311.144834] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing yml_node [ 316.188652] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 319.410368] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 322.552467] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 324.469467] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 16:36:16 EDT 2026 [ 344.252104] Lustre: Unmounted lustre-client [ 375.091191] Key type lgssc unregistered [ 375.462902] LNet: 8433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.473713] LNetError: 8433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 375.486132] LNet: Removed LNI 192.168.204.25@tcp [ 376.248314] Key type .llcrypt unregistered [ 376.251488] Key type ._llcrypt unregistered [ 407.176958] Key type ._llcrypt registered [ 407.179256] Key type .llcrypt registered [ 407.432948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.445161] alg: No test for adler32 (adler32-zlib) [ 408.431505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 408.540286] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 408.579165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 413.589750] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 413.867173] LNet: 9315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.877495] LNetError: 9315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 413.903355] LNet: Removed LNI 192.168.204.25@tcp [ 414.871079] Key type .llcrypt unregistered [ 414.880531] Key type ._llcrypt unregistered [ 422.376684] Key type ._llcrypt registered [ 422.382633] Key type .llcrypt registered [ 424.269878] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:37:55 (1788813475) [ 424.667537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 424.687963] alg: No test for adler32 (adler32-zlib) [ 425.667846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.794817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.823161] LNet: 9723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.856064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.958309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.984596] LNet: 9730:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 434.467462] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:38:06 (1788813486) [ 434.601794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.631406] LNet: 10303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.662120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.804860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 442.936579] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:38:14 (1788813494) [ 443.245862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.281651] LNet: 10897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.320796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.462986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 451.666783] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:38:23 (1788813503) [ 451.935503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.962089] LNet: 11482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.012217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.192869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 462.201746] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:38:33 (1788813513) [ 462.387990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.437341] LNet: 12060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.506978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.738768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 463.660700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 474.511451] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:38:45 (1788813525) [ 474.724742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.796571] LNet: 12653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.873435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.143079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 476.453305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 477.413798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 484.934506] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:38:57 (1788813537) [ 485.150953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.197618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.370899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 494.542696] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:39:05 (1788813545) [ 494.741943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.786635] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.805701] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 494.886926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.987992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 495.051409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 495.416799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 495.454942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 496.063103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 496.110745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 496.774744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 496.812184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 497.377132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 497.443684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 498.184253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 498.239262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 498.864542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 498.902703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 499.492534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 499.544497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 500.244983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 500.281808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 500.835811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 500.870286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 501.412428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 501.443488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 501.799404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 501.869032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 502.438234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 502.485738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 510.121543] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:39:21 (1788813561) [ 510.249240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.330587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.458533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 510.536196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 518.848727] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:39:30 (1788813570) [ 518.970325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.032810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.190321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 519.230965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 528.194552] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:39:39 (1788813579) [ 528.334635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.366344] LNet: 15687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.372269] LNet: 15687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 528.406042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.539925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 528.600990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 537.339886] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:39:48 (1788813588) [ 537.418600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.493736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.614950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 537.650398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 546.604382] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:39:58 (1788813598) [ 546.718549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.816398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.957505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 547.007678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 555.996592] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:40:07 (1788813607) [ 556.507049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.549251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.623421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 556.697915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 564.357040] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:40:16 (1788813616) [ 564.709718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.759833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.885704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.658127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 574.254840] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:40:26 (1788813626) [ 574.598919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.650669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.921635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 575.350222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 584.079725] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:40:35 (1788813635) [ 584.216539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.266855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.456627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.761689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 593.673223] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:40:45 (1788813645) [ 593.826687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.862767] LNet: 19865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.867603] LNet: 19865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 593.887420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.668756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 595.017245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 603.692367] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:40:55 (1788813655) [ 603.854405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.938688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.151221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.219531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 612.544286] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:41:04 (1788813664) [ 612.836602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.943389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.089617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.177627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 622.273267] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:41:14 (1788813674) [ 622.407065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.484703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.629497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 631.660838] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:41:23 (1788813683) [ 631.858523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.939165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.067437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 639.951731] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:41:31 (1788813691) [ 640.259899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.297729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.381316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 640.909902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 649.279554] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:41:40 (1788813700) [ 649.409415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.474357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.602199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 649.638749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 658.484836] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:41:50 (1788813710) [ 659.002397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.093365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.221259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 659.300828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 668.858350] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:42:00 (1788813720) [ 668.979336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.083208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.217786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 669.285632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 678.337452] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:42:09 (1788813729) [ 678.552407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.659709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.818325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 678.877414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 688.033824] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:42:19 (1788813739) [ 688.168124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.268838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.422916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 688.540783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 688.565474] LNetError: 25846:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 688.603513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 688.642242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 696.085675] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:42:27 (1788813747) [ 696.502704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.565964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.726586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XBVP7eQk1t [ 696.871259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XBVP7eQk1t [ 696.956849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XBVP7eQk1t [ 705.600449] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:42:37 (1788813757) [ 705.779409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.903205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.051589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 706.117957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 706.192960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 714.545321] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:42:46 (1788813766) [ 714.662149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.716856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.866259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 714.927923] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 717.562685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 719.356587] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 721.472128] Key type lgssc registered [ 723.179468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 727.859873] LNetError: 27606:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.25@tcp99 has no available nets [ 730.895968] Key type lgssc unregistered [ 731.387269] LNet: 28714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.398238] LNet: 28714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 731.408442] LNetError: 28714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.424891] LNet: Removed LNI 192.168.204.25@tcp [ 732.212612] Key type .llcrypt unregistered [ 732.227176] Key type ._llcrypt unregistered [ 739.874842] Key type ._llcrypt registered [ 739.884370] Key type .llcrypt registered [ 741.157703] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:43:13 (1788813793) [ 741.708430] Key type .llcrypt unregistered [ 741.711383] Key type ._llcrypt unregistered [ 748.915359] Key type ._llcrypt registered [ 748.920504] Key type .llcrypt registered [ 749.142655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.165728] alg: No test for adler32 (adler32-zlib) [ 750.188781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.278458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 750.314499] LNet: 29626:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 750.328165] LNet: 29626:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 750.339245] LNet: 29626:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 750.357776] LNet: Message forwarding will be enabled in 2s [ 753.441682] LNet: Message forwarding enabled [ 754.211266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 754.254269] LNet: Message forwarding will stop in 2s to 6s [ 756.514281] LNet: Message forwarding disabled [ 756.597385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 756.614641] LNet: 29706:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 756.624832] LNet: 29706:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 756.642277] LNet: 29706:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 756.655652] LNet: Message forwarding will be enabled in 2s [ 756.843775] LNet: 29719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.668658] Key type .llcrypt unregistered [ 757.674469] Key type ._llcrypt unregistered [ 773.774488] Key type ._llcrypt registered [ 773.776030] Key type .llcrypt registered [ 775.085665] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:43:47 (1788813827) [ 775.504070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.523702] alg: No test for adler32 (adler32-zlib) [ 776.499596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.630508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 784.483677] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:43:56 (1788813836) [ 784.611903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.643857] LNet: 31113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.704645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.811949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 784.846567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 784.879822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 784.907747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 784.936553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 784.989974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 785.032965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 785.107182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 785.179905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 785.235814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 785.281342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 785.342891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 785.423217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 785.457635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 785.506766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 785.585084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 785.632492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 785.682718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 785.742939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 785.786881] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 793.641319] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:44:05 (1788813845) [ 793.753978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.791453] LNet: 31726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.827448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.996047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 802.686326] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:44:14 (1788813854) [ 805.191580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg425-server [ 807.142549] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:44:19 (1788813859) [ 807.232920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.269852] LNet: 32661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.294714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.399643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.443301] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 808.261522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[26-30]@tcp [ 808.301673] LNet: 32801:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 808.372300] LNetError: 32669:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.26@tcp is being used as a gateway but routing feature is not turned on [ 808.387134] LNetError: 32669:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.26@tcp has gone from up to down [ 816.540739] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:44:28 (1788813868) [ 818.747995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg425-server [ 820.757430] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:44:32 (1788813872) [ 820.874901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.912245] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.920290] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.940273] LNet: Removed LNI 192.168.204.25@tcp [ 820.996896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.089481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.115277] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 822.331429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[26-31/2]@tcp [ 822.367241] LNet: 33854:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 822.375222] LNet: 33854:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 822.414585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[26-31/2]@tcp [ 822.431165] LNetError: 33741:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 822.445614] LNetError: 33741:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 822.456868] LNetError: 33741:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.28@tcp has gone from up to down [ 822.467657] LNetError: 33741:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 830.251922] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:44:42 (1788813882) [ 830.883081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.909061] LNet: 34446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.917468] LNetError: 34446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.959639] LNet: Removed LNI 192.168.204.25@tcp [ 830.990905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.085749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 831.196624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.253200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.402911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 831.489287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.544311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.616590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 831.706861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.747743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.847761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 831.976364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.035935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.085712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 832.164744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.210600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.313958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 832.434674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.470767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.596696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 832.732636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.785098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.868620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 832.962967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.010995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.067688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 833.178971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.233192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.358819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 833.469627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.507994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.590855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 833.682043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 833.729843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.786787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 842.147314] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:44:53 (1788813893) [ 844.324067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg425-server [ 846.734540] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:44:58 (1788813898) [ 849.070573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg425-server [ 851.022765] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:45:03 (1788813903) [ 851.196075] LNet: 35843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.201804] LNet: 35843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 851.778570] Key type .llcrypt unregistered [ 851.783518] Key type ._llcrypt unregistered [ 860.256764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 860.548250] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 860.558727] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.573538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 863.243634] Key type ._llcrypt registered [ 863.246855] Key type .llcrypt registered [ 863.520547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.552597] alg: No test for adler32 (adler32-zlib) [ 864.537910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.603940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.631793] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 865.575432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 865.601728] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.502099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 868.541419] LNet: Removed LNI 10.1.2.3@tcp [ 868.672521] LNet: 36563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.679532] LNetError: 36563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.693459] LNet: Removed LNI 192.168.204.25@tcp [ 869.651390] Key type .llcrypt unregistered [ 869.655587] Key type ._llcrypt unregistered [ 884.993316] Key type ._llcrypt registered [ 884.998021] Key type .llcrypt registered [ 886.573900] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:45:38 (1788813938) [ 887.409283] Key type .llcrypt unregistered [ 887.411075] Key type ._llcrypt unregistered [ 893.679360] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 893.851533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.863389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 893.873890] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 896.650727] Key type ._llcrypt registered [ 896.653596] Key type .llcrypt registered [ 896.914920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 896.944104] alg: No test for adler32 (adler32-zlib) [ 897.851266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.910501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 897.937872] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 898.800945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 898.837966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.128183] LNet: 38055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.138512] LNetError: 38055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 902.146397] LNet: Removed LNI 10.1.2.3@tcp [ 902.163613] LNet: Removed LNI 192.168.204.25@tcp [ 902.935768] Key type .llcrypt unregistered [ 902.937573] Key type ._llcrypt unregistered [ 916.996740] Key type ._llcrypt registered [ 916.999656] Key type .llcrypt registered [ 918.442300] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:46:10 (1788813970) [ 919.087471] Key type .llcrypt unregistered [ 919.091767] Key type ._llcrypt unregistered [ 926.104105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.366398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.376226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.397474] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.046785] Key type ._llcrypt registered [ 929.051982] Key type .llcrypt registered [ 929.379461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.402187] alg: No test for adler32 (adler32-zlib) [ 930.436916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.520391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 930.550838] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 932.965564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 932.986609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.047191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 935.080726] LNet: Removed LNI 10.1.2.3@tcp [ 935.095112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 935.124358] LNet: Removed LNI 10.1.2.31@tcp [ 935.143000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 935.170422] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.825876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 935.865190] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 936.865338] LNet: 39972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.868807] LNetError: 39972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 936.872641] LNet: Removed LNI 10.1.2.31@tcp [ 936.882058] LNet: Removed LNI 10.1.2.3@tcp [ 937.584850] Key type .llcrypt unregistered [ 937.588398] Key type ._llcrypt unregistered [ 952.213644] Key type ._llcrypt registered [ 952.216696] Key type .llcrypt registered [ 953.829209] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:46:45 (1788814005) [ 954.354882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 954.366750] alg: No test for adler32 (adler32-zlib) [ 955.339229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.471358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.538372] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 956.623764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.646724] LNet: 40923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.653104] LNetError: 40923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.674969] LNet: Removed LNI 192.168.204.25@tcp [ 956.703680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.789627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.25@tcp [ 956.844849] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 957.836988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.25@tcp [ 957.904180] LNet: Removed LNI 192.168.204.25@tcp [ 958.039944] LNet: 41058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.895402] Key type .llcrypt unregistered [ 958.900767] Key type ._llcrypt unregistered [ 972.204083] Key type ._llcrypt registered [ 972.206587] Key type .llcrypt registered [ 973.729380] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:47:05 (1788814025) [ 975.903759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg425-server [ 977.981937] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:47:09 (1788814029) [ 978.589429] Key type .llcrypt unregistered [ 978.592473] Key type ._llcrypt unregistered [ 985.884189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.250135] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.276371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.286654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 989.164682] Key type ._llcrypt registered [ 989.167807] Key type .llcrypt registered [ 989.568511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.611574] alg: No test for adler32 (adler32-zlib) [ 990.744688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.848938] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 991.663781] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 994.336948] LNet: 42891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.341588] LNetError: 42891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.349458] LNet: Removed LNI 192.168.204.25@tcp1 [ 994.363958] LNet: Removed LNI 10.1.2.3@tcp2 [ 995.122540] Key type .llcrypt unregistered [ 995.125276] Key type ._llcrypt unregistered [ 1008.806814] Key type ._llcrypt registered [ 1008.809564] Key type .llcrypt registered [ 1010.242320] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:47:42 (1788814062) [ 1010.639614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1010.672152] alg: No test for adler32 (adler32-zlib) [ 1011.627991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.689979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.713587] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1012.961786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.997669] LNet: 43859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.004385] LNetError: 43859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.025700] LNet: Removed LNI 192.168.204.25@tcp [ 1013.053664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.172463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1013.224322] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1021.227404] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:47:52 (1788814072) [ 1021.325593] LNet: 44442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.339583] LNetError: 44442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.356270] LNet: Removed LNI 192.168.204.25@tcp [ 1021.942375] Key type .llcrypt unregistered [ 1021.953523] Key type ._llcrypt unregistered [ 1027.999119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1028.246979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1028.259124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1028.267620] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1030.798765] Key type ._llcrypt registered [ 1030.800648] Key type .llcrypt registered [ 1031.039545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1031.080941] alg: No test for adler32 (adler32-zlib) [ 1032.095833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.164320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1032.221965] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1032.963372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1033.012483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.547054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.603364] LNet: 45184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.620554] LNetError: 45184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.629975] LNet: Removed LNI 10.1.2.3@tcp [ 1035.640538] LNet: Removed LNI 192.168.204.25@tcp [ 1035.664897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.766783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1035.810104] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1035.836438] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1036.453618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.473387] LNet: 45212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.479806] LNetError: 45212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.484880] LNetError: 45212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1036.490659] LNet: Removed LNI 10.1.2.3@tcp [ 1036.503041] LNet: Removed LNI 192.168.204.25@tcp [ 1036.538573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.608155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1036.630943] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1036.634822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.235507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1037.285813] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1038.356588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.378753] LNet: 45374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.385961] LNetError: 45374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.390919] LNetError: 45374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1038.397743] LNet: Removed LNI 10.1.2.3@tcp [ 1038.400129] LNet: Removed LNI 192.168.204.25@tcp [ 1038.418236] LNet: Removed LNI 192.168.204.25@tcp2 [ 1038.458843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.534136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1038.563143] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1038.567749] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1038.572947] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1046.776548] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:48:18 (1788814098) [ 1046.960251] LNet: 45968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.967264] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.981238] LNetError: 45968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1046.995137] LNet: Removed LNI 10.1.2.3@tcp [ 1046.998278] LNet: Removed LNI 192.168.204.25@tcp [ 1047.031252] LNet: Removed LNI 192.168.204.25@tcp2 [ 1047.660690] Key type .llcrypt unregistered [ 1047.663605] Key type ._llcrypt unregistered [ 1054.404867] Key type ._llcrypt registered [ 1054.413840] Key type .llcrypt registered [ 1054.724907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.769790] alg: No test for adler32 (adler32-zlib) [ 1055.758663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.838955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1055.886947] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1057.184685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.207753] LNet: 46363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.218221] LNetError: 46363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.237580] LNet: Removed LNI 192.168.204.25@tcp [ 1057.253657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.304667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1057.326308] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1057.684096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.711607] LNet: Removed LNI 192.168.204.25@tcp [ 1057.733562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.816860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1057.844297] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1058.399840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.434600] LNet: 46423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.442602] LNet: 46423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1058.456675] LNetError: 46423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.466082] LNetError: 46423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1058.483423] LNet: Removed LNI 192.168.204.25@tcp [ 1058.517916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.607794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1058.634922] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1059.340317] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1059.604832] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1059.612546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1059.626730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1061.977902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.997685] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.008711] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.023652] LNet: Removed LNI 192.168.204.25@tcp [ 1062.045484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.109886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1062.133954] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1070.680570] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:48:42 (1788814122) [ 1070.801957] LNet: 47092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.817230] LNetError: 47092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.836371] LNet: Removed LNI 192.168.204.25@tcp [ 1071.639496] Key type .llcrypt unregistered [ 1071.641870] Key type ._llcrypt unregistered [ 1079.291716] Key type ._llcrypt registered [ 1079.293592] Key type .llcrypt registered [ 1079.641753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.698372] alg: No test for adler32 (adler32-zlib) [ 1080.697490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.825207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.859602] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1081.829769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.867258] LNet: 47496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.873608] LNetError: 47496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.900559] LNet: Removed LNI 192.168.204.25@tcp [ 1081.958966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.039636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1082.094198] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1089.916339] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:49:01 (1788814141) [ 1090.051579] LNet: 48081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.063992] LNetError: 48081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.079581] LNet: Removed LNI 192.168.204.25@tcp [ 1090.843636] Key type .llcrypt unregistered [ 1090.848662] Key type ._llcrypt unregistered [ 1097.335786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1097.475939] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1097.484166] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1097.492196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1100.278455] Key type ._llcrypt registered [ 1100.282303] Key type .llcrypt registered [ 1100.573458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.600335] alg: No test for adler32 (adler32-zlib) [ 1101.639977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.749526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1101.789798] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1102.868597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1102.900350] LNet: 48538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.904527] LNetError: 48538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.929366] LNet: Removed LNI 192.168.204.25@tcp [ 1102.967317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.045903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1103.070555] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1111.199329] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:49:23 (1788814163) [ 1111.316440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1111.355836] LNet: 49129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.364645] LNetError: 49129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.391281] LNet: Removed LNI 192.168.204.25@tcp [ 1111.411452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.575341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1118.247689] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:49:30 (1788814170) [ 1118.409392] LNet: 49710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.193420] Key type .llcrypt unregistered [ 1119.196452] Key type ._llcrypt unregistered [ 1125.339666] Key type ._llcrypt registered [ 1125.342483] Key type .llcrypt registered [ 1125.629535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.647268] alg: No test for adler32 (adler32-zlib) [ 1126.622464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.694805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1126.728880] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1127.662937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.683343] LNet: 50113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.687986] LNetError: 50113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.705463] LNet: Removed LNI 192.168.204.25@tcp [ 1127.744422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.824167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1127.869901] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1135.077634] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:49:47 (1788814187) [ 1135.207170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1135.224980] LNet: 50700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.230330] LNetError: 50700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.239939] LNet: Removed LNI 192.168.204.25@tcp [ 1135.273777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.346468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1142.045710] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:49:54 (1788814194) [ 1143.868928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1145.500216] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:49:57 (1788814197) [ 1146.993281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1148.429950] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:50:00 (1788814200) [ 1150.057626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1152.009178] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:50:04 (1788814204) [ 1153.668571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1155.498699] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:50:07 (1788814207) [ 1156.924143] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1158.608908] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:50:10 (1788814210) [ 1158.854901] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.563284] Key type .llcrypt unregistered [ 1159.566339] Key type ._llcrypt unregistered [ 1165.711941] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1165.867140] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1165.870885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1165.873982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1168.559411] Key type ._llcrypt registered [ 1168.561849] Key type .llcrypt registered [ 1168.918751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.936825] alg: No test for adler32 (adler32-zlib) [ 1169.817673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.886971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1169.907330] LNet: Added LNI 192.168.204.25@tcp [8/512/0/180] [ 1169.913553] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1169.916513] LNet: 53041:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1170.036384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1170.095759] LNet: 53060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.100424] LNetError: 53060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1170.107766] LNet: Removed LNI 10.1.2.3@tcp [ 1170.110515] LNetError: 53060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.141484] LNet: Removed LNI 192.168.204.25@tcp [ 1170.167852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.278916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1170.308372] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1170.378310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1170.405763] LNet: 53084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.412804] LNetError: 53084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.433664] LNet: Removed LNI 192.168.204.25@tcp [ 1170.488951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.626749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1170.652092] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1170.659913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1170.667930] LNet: 53092:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1170.771458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1170.788166] LNet: 53110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.794256] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1170.798674] LNet: Removed LNI 10.1.2.3@tcp [ 1170.801347] LNetError: 53110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.812194] LNet: Removed LNI 192.168.204.25@tcp [ 1170.839824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.933316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1170.958963] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1170.962303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1170.964226] LNet: 53118:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1171.140243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.170396] LNet: 53144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.177949] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1171.186507] LNet: Removed LNI 10.1.2.3@tcp [ 1171.190188] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.206760] LNet: Removed LNI 192.168.204.25@tcp [ 1171.246673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.321513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1171.350654] LNet: Added LNI 192.168.204.25@tcp [8/256/0/182] [ 1171.355405] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1171.471575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1171.505350] LNet: 53172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.514718] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1171.522737] LNet: Removed LNI 10.1.2.3@tcp [ 1171.524176] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.545481] LNet: Removed LNI 192.168.204.25@tcp [ 1171.579646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.658927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1171.695726] LNet: Added LNI 192.168.204.25@tcp [8/256/0/182] [ 1171.706653] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1171.711134] LNet: 53180:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1171.822628] LNet: Removed LNI 10.1.2.3@tcp [ 1171.838448] LNet: Removed LNI 192.168.204.25@tcp [ 1172.772964] Key type .llcrypt unregistered [ 1172.775392] Key type ._llcrypt unregistered [ 1183.791116] Key type ._llcrypt registered [ 1183.793391] Key type .llcrypt registered [ 1185.065958] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:50:37 (1788814237) [ 1186.850071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1188.703915] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:50:40 (1788814240) [ 1189.275243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.298270] alg: No test for adler32 (adler32-zlib) [ 1190.229344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.332662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1190.380505] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1191.352982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.25 [ 1191.372610] LNet: 54442:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1191.383768] LNet: Removed LNI 192.168.204.25@tcp [ 1198.064420] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:50:50 (1788814250) [ 1198.422976] LNet: 55013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.432805] LNetError: 55013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.461451] LNet: Removed LNI 192.168.204.25@tcp [ 1199.257418] Key type .llcrypt unregistered [ 1199.263365] Key type ._llcrypt unregistered [ 1205.551621] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1205.845566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1208.535395] Key type ._llcrypt registered [ 1208.541411] Key type .llcrypt registered [ 1208.714605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1208.729384] alg: No test for adler32 (adler32-zlib) [ 1210.184121] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1216.562604] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:51:08 (1788814268) [ 1216.785803] LNet: 55924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.792418] LNetError: 55924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1216.802163] LNet: Removed LNI 10.1.2.3@tcp [ 1217.438992] Key type .llcrypt unregistered [ 1217.444727] Key type ._llcrypt unregistered [ 1224.037629] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1224.371159] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1224.377637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1224.383977] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1227.213709] Key type ._llcrypt registered [ 1227.218322] Key type .llcrypt registered [ 1227.479324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.508233] alg: No test for adler32 (adler32-zlib) [ 1228.532228] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1235.604925] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:51:27 (1788814287) [ 1237.082608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1238.837602] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:51:30 (1788814290) [ 1238.991461] LNet: 57104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.001171] LNetError: 57104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1239.032589] LNet: Removed LNI 10.1.2.3@tcp [ 1239.621637] Key type .llcrypt unregistered [ 1239.623668] Key type ._llcrypt unregistered [ 1245.918944] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1246.224839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1246.231090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1246.238187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1249.045423] Key type ._llcrypt registered [ 1249.048437] Key type .llcrypt registered [ 1249.264717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.280783] alg: No test for adler32 (adler32-zlib) [ 1250.253388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.354042] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1258.483661] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:51:50 (1788814310) [ 1258.977526] LNet: 58333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.984801] LNetError: 58333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1259.001852] LNet: Removed LNI 10.1.2.3@tcp [ 1259.617450] Key type .llcrypt unregistered [ 1259.620428] Key type ._llcrypt unregistered [ 1264.585224] Key type ._llcrypt registered [ 1264.587781] Key type .llcrypt registered [ 1264.765198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.776688] alg: No test for adler32 (adler32-zlib) [ 1265.737550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.778996] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1273.363988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1274.195428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1274.240411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1274.274615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1274.300623] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1274.553707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.592496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1274.626741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.658156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.682993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1274.720858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.749737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.801492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.851818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.872714] LNet: 58769:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.909311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.962174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.023322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.040733] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1275.045380] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1275.091672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.142786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.203125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.244497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.706497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1275.754633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1275.806524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1276.097865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.141559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1276.193164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.241427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.307065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1276.363464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.405868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.478477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.553479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.583832] LNet: 58880:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.620238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.669936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.707359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.756212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.816960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.850640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.922136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.719369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1277.771339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1277.857232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1277.887520] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1277.900335] LNet: Skipped 1 previous similar message [ 1278.178301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.231542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.284789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.316842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.358200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1278.384221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.414341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.442396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.473846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.494940] LNet: 58992:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.532157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.575755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.613868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.668993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.712465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.760702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.792588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.314800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1279.361232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1279.406425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1279.758703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.796355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.837339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.884844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.943601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1279.996481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.039665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.077690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.117675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.158375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.201633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.235488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.287784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.370875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.448726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.507636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.127938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1281.170407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1281.210933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1281.232040] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1281.239522] LNet: Skipped 1 previous similar message [ 1281.502280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.532939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.594786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.628937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1281.660892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1281.687567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1281.711984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1281.742568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1281.785842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1281.801420] LNet: 59214:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1281.806734] LNet: 59214:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1281.826718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1281.853533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1281.894331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.929479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.968500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.999874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.035431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1282.494093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1282.525431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1282.577240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1282.794683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.824202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1282.850928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.896066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.930847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1282.967511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.005177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.043322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.082376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.111647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.166722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.237746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.270272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.339389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.380359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.432353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1283.742583] LNet: 59367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.747590] LNetError: 59367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.764503] LNet: Removed LNI 192.168.204.25@tcp [ 1284.113617] Key type .llcrypt unregistered [ 1284.116444] Key type ._llcrypt unregistered [ 1300.550896] Key type ._llcrypt registered [ 1300.554678] Key type .llcrypt registered [ 1301.679262] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:52:33 (1788814353) [ 1302.424829] Key type .llcrypt unregistered [ 1302.429251] Key type ._llcrypt unregistered [ 1307.034637] Key type ._llcrypt registered [ 1307.038237] Key type .llcrypt registered [ 1307.223479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1307.241623] alg: No test for adler32 (adler32-zlib) [ 1308.126321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1308.156978] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1314.572954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1324.688679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.734377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1324.771551] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1326.134235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1326.167525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 1 [ 1326.206741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1326.283243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 1 [ 1326.327540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_interrupt -r 1 [ 1326.381897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 1 [ 1326.422545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_interrupt -r 1 [ 1326.475650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 1 [ 1326.520509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1326.542554] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1326.551752] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1326.852285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1326.898144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1326.962417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1327.003772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1327.035492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1327.062466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1327.088700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1327.112639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1327.138398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1327.153757] LNet: 60831:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1327.172386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1327.203369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1327.232037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1327.273075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1327.320280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1327.346991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1327.391579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1327.888193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1327.937850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 1 [ 1327.971713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1328.034214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 1 [ 1328.094180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_dropped -r 1 [ 1328.130266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 1 [ 1328.168765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_dropped -r 1 [ 1328.211724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 1 [ 1328.247671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1328.265577] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1328.270622] LNet: Skipped 2 previous similar messages [ 1328.583648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1328.625896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1328.665953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1328.691567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1328.726452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1328.763297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1328.796161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1328.840042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1328.868796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1328.889933] LNet: 60954:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1328.909573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1328.961141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1329.001402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1329.039243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1329.078879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1329.116454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.178224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.760796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1329.805305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 1 [ 1329.847988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1329.903407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 1 [ 1329.956332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_aborted -r 1 [ 1330.012886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 1 [ 1330.069225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_aborted -r 1 [ 1330.110731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 1 [ 1330.149364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1330.171897] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1330.181415] LNet: Skipped 2 previous similar messages [ 1330.441026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.472416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.500647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.555226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.598583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1330.646080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.690481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.728390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.760470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.773348] LNet: 61077:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.795128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.825538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.863711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.911969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.954087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.979130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.027632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1331.452761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1331.484959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 1 [ 1331.520716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1331.560481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 1 [ 1331.605293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_no_route -r 1 [ 1331.647286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 1 [ 1331.692481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_no_route -r 1 [ 1331.745217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 1 [ 1331.801950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1331.966703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.009379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1332.034121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.057914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.090560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1332.122961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.147099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.187383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.252538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.296247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.327459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.355447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.385457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.409609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.430664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.455505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1332.880603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1332.908856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 1 [ 1332.944021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1332.977089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 1 [ 1333.030404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_timeout -r 1 [ 1333.076385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 1 [ 1333.121283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_timeout -r 1 [ 1333.162524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 1 [ 1333.204624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1333.219370] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1333.223694] LNet: Skipped 5 previous similar messages [ 1333.413743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.442256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.476992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.518300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.551816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1333.583213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.622622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.659433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.700648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.733269] LNet: 61324:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.745263] LNet: 61324:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1333.781478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1333.830867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1333.868821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1333.895921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1333.927218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1333.968782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1333.991941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.376762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1334.407932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_error -r 1 [ 1334.437354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1334.469140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_error -r 1 [ 1334.501857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_error -r 1 [ 1334.534406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_error -r 1 [ 1334.571715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_error -r 1 [ 1334.623553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_error -r 1 [ 1334.663144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1334.895649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.936104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.007553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.062846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.110391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1335.137784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.167543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.203276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1335.230960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1335.262348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1335.319188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1335.353558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1335.405316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.444869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.497554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.560738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.439544] LNet: 61523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.448621] LNetError: 61523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.463400] LNet: Removed LNI 192.168.204.25@tcp [ 1342.503726] LNet: Removed LNI 192.168.204.25@tcp1 [ 1342.916748] Key type .llcrypt unregistered [ 1342.920427] Key type ._llcrypt unregistered [ 1357.002205] Key type ._llcrypt registered [ 1357.004391] Key type .llcrypt registered [ 1358.336669] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:53:30 (1788814410) [ 1359.120555] Key type .llcrypt unregistered [ 1359.122668] Key type ._llcrypt unregistered [ 1364.086045] Key type ._llcrypt registered [ 1364.088456] Key type .llcrypt registered [ 1364.253028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.266787] alg: No test for adler32 (adler32-zlib) [ 1365.157834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1365.180457] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1372.446653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1373.123275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1373.167980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1373.215320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1373.245286] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1373.506852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1373.536685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1373.576789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1373.627574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1373.664131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1373.694717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1373.722148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1373.760246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1373.796893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1373.817445] LNet: 62770:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1373.834950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1373.860355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1373.889311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1373.928595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1373.972742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1374.023097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1374.066953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1374.532510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1374.576157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1374.620498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1374.643453] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1374.851748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1374.878413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1374.908447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1374.955505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1374.989541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1375.026385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1375.076379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1375.129257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1375.171342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1375.200937] LNet: 62881:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1375.220161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1375.247942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1375.291669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.344401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1375.386393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.424875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.462215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1376.002896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1376.042740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1376.099771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1376.132221] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1376.373432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.410605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.444745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.473603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1376.514325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1376.546659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1376.589749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1376.622679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1376.654814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1376.672350] LNet: 62992:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.694855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.755391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.798950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.841508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.895652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.934044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1376.966335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1377.218653] LNet: 63033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.224568] LNetError: 63033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.240991] LNet: Removed LNI 192.168.204.25@tcp [ 1377.620533] Key type .llcrypt unregistered [ 1377.623318] Key type ._llcrypt unregistered [ 1394.474603] Key type ._llcrypt registered [ 1394.476747] Key type .llcrypt registered [ 1395.762526] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:54:07 (1788814447) [ 1396.685476] Key type .llcrypt unregistered [ 1396.692140] Key type ._llcrypt unregistered [ 1402.492508] Key type ._llcrypt registered [ 1402.497035] Key type .llcrypt registered [ 1402.728795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1402.756497] alg: No test for adler32 (adler32-zlib) [ 1403.651676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1403.692700] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1410.759536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1418.921441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.965861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1418.991490] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1420.194713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1420.224639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 1 [ 1420.261661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1420.319623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 1 [ 1420.357480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_dropped -r 1 [ 1420.388568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 1 [ 1420.419618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_dropped -r 1 [ 1420.473288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 1 [ 1420.524712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1420.562155] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1420.568826] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1420.804559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1420.836601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1420.863954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1420.891856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1420.930932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1420.962442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1420.986473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1421.016852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1421.050337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1421.066860] LNet: 64498:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1421.087448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1421.117295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1421.155534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1421.183293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1421.238373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.315900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1421.377689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1421.427602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.956783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1421.990774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_error -r 1 [ 1422.034727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1422.086436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_error -r 1 [ 1422.138963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_error -r 1 [ 1422.180864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_error -r 1 [ 1422.248723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_error -r 1 [ 1422.287783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_error -r 1 [ 1422.318394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1422.346266] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1422.351406] LNet: Skipped 2 previous similar messages [ 1422.622325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1422.649446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1422.685688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1422.717520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1422.742423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1422.770199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1422.800385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1422.842449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1422.882471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1422.901394] LNet: 64623:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1422.923205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1422.971564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1423.011792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.045918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1423.071221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1423.125470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1423.168375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1423.213540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1423.760110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1423.818353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_timeout -r 1 [ 1423.867334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1423.906747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_timeout -r 1 [ 1423.947079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_timeout -r 1 [ 1423.987524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_timeout -r 1 [ 1424.044771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_timeout -r 1 [ 1424.095826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_timeout -r 1 [ 1424.155309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.125@tcp [ 1424.182992] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1424.375249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1424.404041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1424.432653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1424.460715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1424.490662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1424.519519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1424.544806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1424.578491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1424.611734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1424.634308] LNet: 64749:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1424.656657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1424.689088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1424.727532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1424.758906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1424.792243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1424.826387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1424.866511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1424.908473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1429.593184] LNet: 64827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.601462] LNetError: 64827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.611756] LNet: Removed LNI 192.168.204.25@tcp [ 1429.628643] LNet: Removed LNI 192.168.204.25@tcp1 [ 1430.041853] Key type .llcrypt unregistered [ 1430.045033] Key type ._llcrypt unregistered [ 1445.482530] Key type ._llcrypt registered [ 1445.487415] Key type .llcrypt registered [ 1446.577371] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:54:58 (1788814498) [ 1448.196615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1449.784640] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:55:02 (1788814502) [ 1450.516541] Key type .llcrypt unregistered [ 1450.519507] Key type ._llcrypt unregistered [ 1456.741585] Key type ._llcrypt registered [ 1456.745768] Key type .llcrypt registered [ 1457.004612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1457.022191] alg: No test for adler32 (adler32-zlib) [ 1457.979678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1458.038595] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1464.664368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1465.245912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1465.297200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1465.352622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1465.373647] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1465.545611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1465.584712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1465.610932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1465.633930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1465.664757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1465.693319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1465.720983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1465.749713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1465.774337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1465.790922] LNet: 66367:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1465.809274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1465.842521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1465.874555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.906307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1465.940437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1465.968537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1466.188931] LNet: 66407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.195530] LNetError: 66407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.216799] LNet: Removed LNI 192.168.204.25@tcp [ 1466.513716] Key type .llcrypt unregistered [ 1466.516888] Key type ._llcrypt unregistered [ 1482.480481] Key type ._llcrypt registered [ 1482.482315] Key type .llcrypt registered [ 1482.798577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1482.815474] alg: No test for adler32 (adler32-zlib) [ 1483.727115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1483.766311] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1490.652687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1498.961509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.021151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.050357] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1500.390246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1500.430833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e network_timeout -r 1 [ 1500.476185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1500.528062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e network_timeout -r 1 [ 1500.590451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e network_timeout -r 1 [ 1500.649420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e network_timeout -r 1 [ 1500.711365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e network_timeout -r 1 [ 1500.765833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e network_timeout -r 1 [ 1500.817185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1500.852996] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1501.086818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1501.117641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1501.147752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1501.178563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1501.217484] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1501.218523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1501.221934] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1501.264450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1501.298175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1501.333379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1501.369377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1501.396810] LNet: 67283:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1501.419920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1501.455377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1501.488356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1501.526306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1501.562882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1501.605853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1505.987633] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1505.998068] LNet: Skipped 2 previous similar messages [ 1506.245922] LNet: 67357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.259449] LNetError: 67357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.267468] LNet: Removed LNI 192.168.204.25@tcp [ 1506.275298] LNet: Removed LNI 192.168.204.25@tcp1 [ 1506.604513] Key type .llcrypt unregistered [ 1506.607360] Key type ._llcrypt unregistered [ 1519.971189] Key type ._llcrypt registered [ 1519.974560] Key type .llcrypt registered [ 1521.224455] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:56:13 (1788814573) [ 1521.666579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.684427] alg: No test for adler32 (adler32-zlib) [ 1522.580967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.622221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.653079] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1523.308477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.325908] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1524.110929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1524.173407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1524.233448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1524.274901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1524.325422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1524.369207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1524.421292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1524.461389] LNetError: 68454:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1524.639282] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1524.639708] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1526.687813] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1526.891841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1530.783533] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1531.501764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1531.544307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1531.599774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.628680] LNet: 68561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.634808] LNetError: 68561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.703413] LNet: Removed LNI 192.168.204.25@tcp [ 1532.707168] LNetError: 68561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.733743] LNet: Removed LNI 192.168.204.25@tcp1 [ 1532.766423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.859400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1532.891698] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1533.602411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1533.633938] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1534.328421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1534.359861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1534.439716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1534.484675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1534.512665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1534.553184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1534.576862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1534.620524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1534.648090] LNetError: 68834:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1534.882550] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1537.109908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1541.023404] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1541.033617] LNet: Skipped 1 previous similar message [ 1546.097493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1546.128978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1546.159499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1552.925943] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:56:45 (1788814605) [ 1553.158187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.181179] LNet: 69552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.187382] LNetError: 69552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.197905] LNet: Removed LNI 192.168.204.25@tcp [ 1553.208345] LNet: Removed LNI 192.168.204.25@tcp1 [ 1553.244496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.315518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.345276] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1554.042937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.062232] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1554.780057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1554.834914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1554.869569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1554.903129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1554.935776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1554.971139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1555.007028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 0 [ 1555.113875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1555.295159] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.25@tcp [ 1556.319427] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1556.323895] LNet: Skipped 1 previous similar message [ 1571.312850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1571.418092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1571.441934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1571.463603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 500 [ 1571.533724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1578.035105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1578.079375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.101221] LNet: 70052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.108521] LNetError: 70052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.113780] LNetError: 70052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1578.119385] LNet: Removed LNI 192.168.204.25@tcp [ 1578.134525] LNet: Removed LNI 192.168.204.25@tcp1 [ 1578.159283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.216904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.244693] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1578.894787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1578.919493] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1579.621080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1579.650096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.705488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1579.748922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1579.799894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1579.837466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1579.888386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1579.928605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.25@tcp --health 0 [ 1580.255564] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1580.268924] LNet: Skipped 3 previous similar messages [ 1581.237912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1591.270867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1591.301657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1591.335481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1596.390957] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:57:28 (1788814648) [ 1596.847852] LNet: 71099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.853816] LNetError: 71099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.856296] LNetError: 71099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1596.863411] LNet: Removed LNI 192.168.204.25@tcp [ 1596.872218] LNet: Removed LNI 192.168.204.25@tcp1 [ 1597.366858] Key type .llcrypt unregistered [ 1597.370425] Key type ._llcrypt unregistered [ 1601.661753] Key type ._llcrypt registered [ 1601.664288] Key type .llcrypt registered [ 1601.842666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1601.858595] alg: No test for adler32 (adler32-zlib) [ 1602.774627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1602.800343] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1610.380222] Lustre: DEBUG MARKER: Initial discovery [ 1610.486341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1612.255986] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1612.289739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1612.319608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1612.343432] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1612.361344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1612.384577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.125@tcp -r 1 -m GET -l 3 [ 1612.424721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.125@tcp -r 1 -m GET -e local_error [ 1612.460425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.125@tcp -r 1 -m PUT -l 6 [ 1613.092168] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1613.481633] Lustre: DEBUG MARKER: Force oleg425-client.virtnet to discover 192.168.204.125@tcp (in background) [ 1613.621120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.125@tcp [ 1614.753907] Lustre: DEBUG MARKER: Force oleg425-server to discover 192.168.204.25@tcp [ 1616.449982] Lustre: DEBUG MARKER: Wait for 71858 [ 1616.674377] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1617.708634] Lustre: DEBUG MARKER: Finished wait on 71858 [ 1617.861948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1617.890349] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1617.903463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1618.114449] LNet: 72307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.122201] LNetError: 72307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.145199] LNet: Removed LNI 192.168.204.25@tcp [ 1618.436465] Key type .llcrypt unregistered [ 1618.439203] Key type ._llcrypt unregistered [ 1630.703468] Key type ._llcrypt registered [ 1630.705759] Key type .llcrypt registered [ 1631.693915] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:58:04 (1788814684) [ 1632.051307] Key type .llcrypt unregistered [ 1632.053275] Key type ._llcrypt unregistered [ 1637.245312] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1637.357188] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1637.366769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1637.380715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1639.838189] Key type ._llcrypt registered [ 1639.840528] Key type .llcrypt registered [ 1640.010021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.039366] alg: No test for adler32 (adler32-zlib) [ 1640.927071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.972494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.996102] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1641.521543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1641.537239] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1647.734786] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:58:20 (1788814700) [ 1647.804388] LNet: 74374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.815193] LNetError: 74374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1647.821697] LNet: Removed LNI 10.1.2.3@tcp [ 1647.829940] LNet: Removed LNI 192.168.204.25@tcp [ 1648.322222] Key type .llcrypt unregistered [ 1648.325594] Key type ._llcrypt unregistered [ 1653.396458] Key type ._llcrypt registered [ 1653.400651] Key type .llcrypt registered [ 1653.572712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1653.587201] alg: No test for adler32 (adler32-zlib) [ 1654.519240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.568876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.597479] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1655.222516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1655.249881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1656.155823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.25@tcp [ 1657.247769] LNetError: Unexpected error -22 connecting to 192.168.204.25@tcp at host 192.168.204.25:988 [ 1661.103456] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:58:33 (1788814713) [ 1661.207894] LNet: 75505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.216548] LNetError: 75505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1661.224802] LNet: Removed LNI 10.1.2.3@tcp [ 1661.235213] LNet: Removed LNI 192.168.204.25@tcp [ 1661.718105] Key type .llcrypt unregistered [ 1661.723036] Key type ._llcrypt unregistered [ 1666.217095] Key type ._llcrypt registered [ 1666.220065] Key type .llcrypt registered [ 1666.337660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1666.355147] alg: No test for adler32 (adler32-zlib) [ 1667.246403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.295756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1667.322274] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1667.837502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1667.857979] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 1668.573543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.25@tcp1 --nid 192.168.204.25@tcp2 [ 1671.830610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp1 192.168.204.25@tcp2 [ 1671.852758] LNetError: 76341:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.25@tcp1 to 192.168.204.25@tcp2 - Route table may be misconfigured [ 1671.867985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp2 192.168.204.25@tcp1 [ 1677.270386] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:58:49 (1788814729) [ 1677.537988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.566543] LNet: 76914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.582067] LNetError: 76914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.590718] LNet: Removed LNI 192.168.204.25@tcp1 [ 1678.631321] LNet: Removed LNI 192.168.204.25@tcp2 [ 1678.656712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.709445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.735093] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1679.308218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1679.334608] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1679.995689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 1680.039425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.25@tcp [ 1680.070358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp1 192.168.204.25@tcp1 [ 1680.101487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.25@tcp -d 192.168.204.25@tcp -e network_timeout [ 1680.131261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -e network_timeout [ 1680.163191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -e network_timeout [ 1680.196933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -e network_timeout [ 1680.246556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1680.272748] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1680.293433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1680.379075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1680.486896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1680.739651] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1685.623842] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:58:57 (1788814737) [ 1685.700357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.731311] LNet: 77771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.741661] LNetError: 77771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.751473] LNetError: 77771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1685.768118] LNet: Removed LNI 192.168.204.25@tcp [ 1686.823086] LNet: Removed LNI 192.168.204.25@tcp1 [ 1686.861238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.991430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1687.207958] LNet: 77813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.504684] Key type .llcrypt unregistered [ 1687.506848] Key type ._llcrypt unregistered [ 1695.645815] Key type ._llcrypt registered [ 1695.647343] Key type .llcrypt registered [ 1696.557478] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:59:09 (1788814749) [ 1696.786600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1696.800717] alg: No test for adler32 (adler32-zlib) [ 1697.679770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.713386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1699.959127] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:59:12 (1788814752) [ 1700.085262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.110539] LNet: 78907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.125568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.166226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.180176] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1700.787875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1700.820397] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1701.460827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1701.492547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp1 [ 1701.519976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp1 [ 1706.434664] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:59:18 (1788814758) [ 1707.975874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg425-server [ 1709.657514] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:59:21 (1788814761) [ 1711.307445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg425-server [ 1712.738364] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:59:25 (1788814765) [ 1714.758025] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg425-server [ 1716.058318] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:59:28 (1788814768) [ 1717.568529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg425-server [ 1718.944265] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:59:31 (1788814771) [ 1720.587652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg425-server [ 1722.162154] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:59:34 (1788814774) [ 1723.464382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg425-server [ 1724.906912] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:59:37 (1788814777) [ 1726.356728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg425-server [ 1727.695849] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:59:40 (1788814780) [ 1729.109420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg425-server [ 1730.637025] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:59:42 (1788814782) [ 1732.262909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg425-server [ 1733.706299] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:59:46 (1788814786) [ 1733.874191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.902377] LNet: 82784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.907694] LNetError: 82784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.916419] LNet: Removed LNI 192.168.204.25@tcp [ 1734.951372] LNet: Removed LNI 192.168.204.25@tcp1 [ 1734.980637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.037610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.068356] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1735.678168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1735.739253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1736.981216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.002327] LNet: 82940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.010821] LNetError: 82940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.018268] LNetError: 82940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1737.032716] LNet: Removed LNI 192.168.204.25@tcp [ 1737.062486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.124685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1737.143291] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1737.712465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1737.854620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.917941] LNet: Removed LNI 192.168.204.25@tcp [ 1738.939423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.002700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.029253] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1739.572776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1739.654772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1739.822592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.841631] LNet: 83233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.850588] LNet: 83233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1739.860877] LNetError: 83233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.870441] LNetError: 83233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1739.889115] LNet: Removed LNI 192.168.204.25@tcp [ 1739.907104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.958171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1739.987331] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1740.611842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1740.778485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.862913] LNet: Removed LNI 192.168.204.25@tcp [ 1741.881292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.943295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.978082] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1742.514792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1742.588114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1742.793702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.816904] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.821140] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1742.839181] LNet: Removed LNI 192.168.204.25@tcp [ 1742.858184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.911759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1742.933259] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1743.637492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1743.834801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.854993] LNet: 83670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.861098] LNet: 83670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1744.933139] LNet: Removed LNI 192.168.204.25@tcp [ 1744.965191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.013414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1745.032095] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1745.568959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1745.646804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1746.931393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.956596] LNetError: 83827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.963978] LNetError: 83827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1746.972770] LNet: Removed LNI 192.168.204.25@tcp [ 1746.990659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.044914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1747.066414] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1747.651841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1748.917546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.991550] LNet: Removed LNI 192.168.204.25@tcp [ 1750.018849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.084545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.108063] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1750.655217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1750.731125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1750.997243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.047613] LNet: Removed LNI 192.168.204.25@tcp [ 1752.094239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.159958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1752.206415] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1752.814252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1752.991865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.006424] LNet: 84273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.011127] LNet: 84273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1754.086141] LNet: Removed LNI 192.168.204.25@tcp [ 1754.105772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.162466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.191027] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1754.791225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1754.855478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1755.053986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.074408] LNetError: 84421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.079403] LNetError: 84421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1756.135677] LNet: Removed LNI 192.168.204.25@tcp [ 1756.153659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.193944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1756.212562] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1756.793072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1757.033827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.123577] LNet: Removed LNI 192.168.204.25@tcp [ 1758.145708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.186842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.212755] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1758.797565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1758.845882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.882780] LNet: Removed LNI 192.168.204.25@tcp [ 1758.908692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.961940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.979694] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1759.578507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1759.625430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.645734] LNet: Removed LNI 192.168.204.25@tcp [ 1759.659659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.704979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1759.799313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.848914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.895792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1759.971468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.007225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.042613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1760.104621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.133227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.190955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1760.227616] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1760.811899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.835525] LNet: Removed LNI 192.168.204.25@tcp [ 1760.855475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.923770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1766.091716] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:00:18 (1788814818) [ 1766.261222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.301273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.350870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1766.371796] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1767.269392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.290703] LNet: Removed LNI 192.168.204.25@tcp231 [ 1767.309579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.352944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1767.372721] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1767.376975] LNet: 85732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.789978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1767.825717] LNet: Removed LNI 192.168.204.25@tcp231 [ 1767.854562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1767.892194] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1768.750443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.778785] LNet: Removed LNI 192.168.204.25@tcp231 [ 1768.808835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.866197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1768.898035] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/0] [ 1768.905701] LNet: 85885:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.440694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1769.493829] LNet: Removed LNI 192.168.204.25@tcp231 [ 1769.513438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1769.539953] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/0] [ 1770.420622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.449567] LNet: 86028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.461366] LNet: 86028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1770.483074] LNet: Removed LNI 192.168.204.25@tcp231 [ 1770.501662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.571722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1770.605685] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/60] [ 1770.609573] LNet: 86036:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1770.962526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1770.987654] LNet: Removed LNI 192.168.204.25@tcp231 [ 1771.001649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1771.024995] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/60] [ 1771.994111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.015349] LNetError: 86179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.019614] LNetError: 86179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1772.033849] LNet: Removed LNI 192.168.204.25@tcp231 [ 1772.054846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.117059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.146614] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1772.531459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1772.563582] LNet: Removed LNI 192.168.204.25@tcp231 [ 1772.578659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1772.605060] LNet: Added LNI 192.168.204.25@tcp231 [8/256/0/180] [ 1778.721860] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:00:31 (1788814831) [ 1778.836461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.860970] LNet: Removed LNI 192.168.204.25@tcp231 [ 1778.891142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.946371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1778.958062] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1779.387359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1779.412785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1779.438379] LNet: Removed LNI 192.168.204.25@tcp232 [ 1779.456457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1779.475919] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1779.933987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1779.959741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1779.985571] LNet: Removed LNI 192.168.204.25@tcp232 [ 1780.000855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1780.016324] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1780.495875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1780.525674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1780.551439] LNet: Removed LNI 192.168.204.25@tcp232 [ 1780.561301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1780.581695] LNet: Added LNI 192.168.204.25@tcp232 [8/256/0/180] [ 1781.078216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1781.102668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1781.123089] LNet: Removed LNI 192.168.204.25@tcp232 [ 1781.134349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1785.710506] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:00:38 (1788814838) [ 1786.817822] Key type .llcrypt unregistered [ 1786.820707] Key type ._llcrypt unregistered [ 1792.302362] Key type ._llcrypt registered [ 1792.303770] Key type .llcrypt registered [ 1792.454921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1792.466762] alg: No test for adler32 (adler32-zlib) [ 1793.409128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1793.432095] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1800.322251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1809.103787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.155710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1809.201327] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1810.573295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_interrupt -r 2 [ 1810.617129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 2 [ 1810.660859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_interrupt -r 2 [ 1810.708755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 2 [ 1810.745627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_interrupt -r 2 [ 1810.783719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_interrupt -r 2 [ 1810.837751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_interrupt -r 2 [ 1810.894069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_interrupt -r 2 [ 1810.944531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1810.981664] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1810.992180] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1811.203545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1811.228758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1811.254441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.278485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.327895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1811.375581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.418640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1811.462447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1811.525661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1811.563747] LNet: 88699:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1811.582148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1811.614799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1811.658602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1811.700309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1811.749982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.790401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.401486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_dropped -r 2 [ 1812.453378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 2 [ 1812.499515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_dropped -r 2 [ 1812.543377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 2 [ 1812.590137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_dropped -r 2 [ 1812.627889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_dropped -r 2 [ 1812.667463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_dropped -r 2 [ 1812.730237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_dropped -r 2 [ 1812.774161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1812.794643] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1813.023836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.058838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1813.095874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.131973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.172931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1813.200966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.228527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.255901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.285372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.302850] LNet: 88821:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.317243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.348794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.379542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.403955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.444266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.489151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.046282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_aborted -r 2 [ 1814.089153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 2 [ 1814.123346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_aborted -r 2 [ 1814.153781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 2 [ 1814.188306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_aborted -r 2 [ 1814.222763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_aborted -r 2 [ 1814.259296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_aborted -r 2 [ 1814.288740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_aborted -r 2 [ 1814.321595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1814.356086] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1814.611745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1814.645476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1814.694715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1814.733836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1814.765757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1814.797299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1814.835678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1814.873922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1814.918535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1814.937454] LNet: 88944:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.953624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.983822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1815.020728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1815.049551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1815.079552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1815.109808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1815.566233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_no_route -r 2 [ 1815.605333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 2 [ 1815.641222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_no_route -r 2 [ 1815.679833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 2 [ 1815.721970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_no_route -r 2 [ 1815.772663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_no_route -r 2 [ 1815.811686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_no_route -r 2 [ 1815.847709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_no_route -r 2 [ 1815.883555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1816.088649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1816.125304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1816.154970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1816.184312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1816.224942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1816.283974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1816.326381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1816.375618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1816.404698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1816.431231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1816.463726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1816.505849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1816.557602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1816.592814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1816.642921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1817.101046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e local_timeout -r 2 [ 1817.135518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 2 [ 1817.158627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e local_timeout -r 2 [ 1817.204390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 2 [ 1817.253964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e local_timeout -r 2 [ 1817.300215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e local_timeout -r 2 [ 1817.349439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e local_timeout -r 2 [ 1817.385865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e local_timeout -r 2 [ 1817.426435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1817.446954] LNet: There was an unexpected network error while writing to 192.168.204.125: rc = -22 [ 1817.454043] LNet: Skipped 1 previous similar message [ 1817.660339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1817.688561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1817.725503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1817.761855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1817.809761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1817.836193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1817.881491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1817.924873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1817.950780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1817.969919] LNet: 89188:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1817.978655] LNet: 89188:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1818.002715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1818.047732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1818.088852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1818.119691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1818.155573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1818.193924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.727692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp -m GET -e remote_dropped -r 2 [ 1818.782167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.25@tcp -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 2 [ 1818.817549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp -m GET -e remote_dropped -r 2 [ 1818.851883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 2 [ 1818.884778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp -m GET -e remote_dropped -r 2 [ 1818.916613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.125@tcp1 -m GET -e remote_dropped -r 2 [ 1818.946772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp -m GET -e remote_dropped -r 2 [ 1818.988611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.25@tcp1 -d 192.168.204.25@tcp1 -m GET -e remote_dropped -r 2 [ 1819.027973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.25@tcp 192.168.204.125@tcp [ 1819.051788] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 1819.236847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.256692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.276851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.300311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.326524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1819.354437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.382125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.409467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.438912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.467588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.495419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.524979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.550428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.580879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.612135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1824.833035] LNet: 89385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.840497] LNetError: 89385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.846028] LNet: Removed LNI 192.168.204.25@tcp [ 1824.862687] LNet: Removed LNI 192.168.204.25@tcp1 [ 1825.367676] Key type .llcrypt unregistered [ 1825.376504] Key type ._llcrypt unregistered [ 1840.709317] Key type ._llcrypt registered [ 1840.713498] Key type .llcrypt registered [ 1842.119314] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:01:34 (1788814894) [ 1842.508874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1842.528968] alg: No test for adler32 (adler32-zlib) [ 1843.420808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.493959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1843.519408] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1844.270768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1844.311954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.25@tcp [ 1844.338484] LNetError: 90362:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1844.431604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.25@tcp [ 1844.578466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1850.352565] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:01:42 (1788814902) [ 1850.444213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.469929] LNet: 90945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.479561] LNetError: 90945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.495720] LNet: Removed LNI 192.168.204.25@tcp [ 1850.529573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.603819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.625484] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1851.390813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1851.409238] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 1852.443395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.25@tcp [ 1852.472845] LNetError: 91217:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.25@tcp to recovery [ 1852.639482] LNet: 1 local NIs in recovery (showing 1): 192.168.204.25@tcp [ 1852.642110] LNet: There was an unexpected network error while writing to 192.168.204.25: rc = -22 [ 1853.753089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.25@tcp [ 1860.645667] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:01:52 (1788814912) [ 1862.699424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg425-server [ 1864.341355] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:01:56 (1788814916) [ 1865.934457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg425-server [ 1867.475188] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:01:59 (1788814919) [ 1869.227845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg425-server [ 1870.803655] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:02:03 (1788814923) [ 1870.889690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.909708] LNet: 92833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.920745] LNetError: 92833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.925167] LNet: Removed LNI 192.168.204.25@tcp [ 1870.936300] LNet: Removed LNI 192.168.204.25@tcp1 [ 1870.965825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.020482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.039751] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1871.759532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1871.822669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1871.877277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1872.024369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1872.081265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1872.161877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1878.136144] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:02:10 (1788814930) [ 1878.222038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.249105] LNet: 93563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.255927] LNetError: 93563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.264368] LNetError: 93563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1878.287148] LNet: Removed LNI 192.168.204.25@tcp [ 1878.335064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.402175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.423098] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1884.760251] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:02:17 (1788814937) [ 1884.903484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.921932] LNet: 94273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.926356] LNetError: 94273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.942286] LNet: Removed LNI 192.168.204.25@tcp [ 1884.957366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.997612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.020188] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1891.513345] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:02:23 (1788814943) [ 1892.971626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1894.526683] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:02:26 (1788814946) [ 1894.997921] LNet: 95307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.008332] LNetError: 95307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.024574] LNet: Removed LNI 192.168.204.25@tcp [ 1895.612308] Key type .llcrypt unregistered [ 1895.614223] Key type ._llcrypt unregistered [ 1901.103562] Key type ._llcrypt registered [ 1901.106815] Key type .llcrypt registered [ 1901.276245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1901.298090] alg: No test for adler32 (adler32-zlib) [ 1902.210295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1902.244045] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1908.667955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 1912.897447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.125@tcp [ 1913.271852] LNet: 95721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.280526] LNetError: 95721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.292145] LNet: Removed LNI 192.168.204.25@tcp [ 1913.761599] Key type .llcrypt unregistered [ 1913.765121] Key type ._llcrypt unregistered [ 1923.370265] Key type ._llcrypt registered [ 1923.372753] Key type .llcrypt registered [ 1924.410476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1926.050475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1927.544940] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:02:59 (1788814979) [ 1928.868367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1930.539120] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:03:02 (1788814982) [ 1932.050810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg425-server [ 1933.535474] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:03:05 (1788814985) [ 1935.468700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg425-server [ 1936.918311] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:03:09 (1788814989) [ 1938.498742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.537709] alg: No test for adler32 (adler32-zlib) [ 1939.640452] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1939.896229] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1941.559229] Key type lgssc registered [ 1942.741696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1959.965716] Key type lgssc unregistered [ 1960.187088] LNet: 98893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.193215] LNetError: 98893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.206561] LNet: Removed LNI 192.168.204.25@tcp [ 1960.837586] Key type .llcrypt unregistered [ 1960.840716] Key type ._llcrypt unregistered [ 1972.864975] Key type ._llcrypt registered [ 1972.867814] Key type .llcrypt registered [ 1973.757286] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:03:46 (1788815026) [ 1974.201222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.215816] alg: No test for adler32 (adler32-zlib) [ 1975.085708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.142686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1975.188273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1975.227044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1975.262049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1980.155700] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:03:52 (1788815032) [ 1980.453150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.484713] LNet: 100305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.530110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.581394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1980.604874] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1981.297494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.320752] LNet: 100449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.330580] LNetError: 100449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.347873] LNet: Removed LNI 192.168.204.25@tcp [ 1981.368488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.423914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1981.444086] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1981.829431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1981.855221] LNetError: 100477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.870790] LNet: Removed LNI 192.168.204.25@tcp [ 1981.892777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.945441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1981.973548] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1987.141443] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:03:59 (1788815039) [ 1987.292514] LNet: 101066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1987.297194] LNet: 101066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1987.305070] LNetError: 101066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1987.320248] LNet: Removed LNI 192.168.204.25@tcp [ 1987.761500] Key type .llcrypt unregistered [ 1987.765391] Key type ._llcrypt unregistered [ 1992.682472] Key type ._llcrypt registered [ 1992.684373] Key type .llcrypt registered [ 1992.833994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.852307] alg: No test for adler32 (adler32-zlib) [ 1993.749333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.843949] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 1993.845820] LNet: *** cfs_fail_loc=e003, val=100*** [ 1994.489632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 1994.510855] LNet: *** cfs_fail_loc=e003, val=100*** [ 1994.516333] LNet: Skipped 2 previous similar messages [ 1994.527711] Ping reply received with mapping: [ 1994.531399] found metadata reply NI[192.168.204.25@tcp] [ 1994.533542] 19 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1994.537522] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1994.540451] 5a 5a 5a 5a 5a 5a 5a 5a [ 1994.636746] LNet: 101462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.642180] LNetError: 101462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.658189] LNet: Removed LNI 192.168.204.25@tcp [ 1995.158410] Key type .llcrypt unregistered [ 1995.161109] Key type ._llcrypt unregistered [ 2004.036914] Key type ._llcrypt registered [ 2004.040248] Key type .llcrypt registered [ 2005.016819] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:04:17 (1788815057) [ 2005.442660] Key type .llcrypt unregistered [ 2005.450454] Key type ._llcrypt unregistered [ 2009.860738] Key type ._llcrypt registered [ 2009.862744] Key type .llcrypt registered [ 2010.028316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.049521] alg: No test for adler32 (adler32-zlib) [ 2011.056322] LNetError: 102498:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2011.375620] Key type .llcrypt unregistered [ 2011.377733] Key type ._llcrypt unregistered [ 2015.361469] Key type ._llcrypt registered [ 2015.364858] Key type .llcrypt registered [ 2016.349907] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:04:28 (1788815068) [ 2016.776417] Key type .llcrypt unregistered [ 2016.780240] Key type ._llcrypt unregistered [ 2021.403684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2021.409429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2023.819387] Key type ._llcrypt registered [ 2023.825082] Key type .llcrypt registered [ 2023.979777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2023.992792] alg: No test for adler32 (adler32-zlib) [ 2024.891679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.955638] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2033.660644] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:04:45 (1788815085) [ 2033.787904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.832220] LNet: 104265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.841139] LNetError: 104265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2033.865953] LNet: Removed LNI 10.1.2.3@tcp [ 2033.898678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.770779] Key type .llcrypt unregistered [ 2034.773472] Key type ._llcrypt unregistered [ 2039.970349] Key type ._llcrypt registered [ 2039.972288] Key type .llcrypt registered [ 2040.147490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.169971] alg: No test for adler32 (adler32-zlib) [ 2041.050343] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2041.106157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2041.235868] LNet: 104557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.246384] LNetError: 104557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.260550] LNet: Removed LNI 192.168.204.25@tcp [ 2041.748387] Key type .llcrypt unregistered [ 2041.751431] Key type ._llcrypt unregistered [ 2050.112331] Key type ._llcrypt registered [ 2050.114300] Key type .llcrypt registered [ 2051.113910] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:05:03 (1788815103) [ 2051.799288] Key type .llcrypt unregistered [ 2051.801413] Key type ._llcrypt unregistered [ 2057.367444] Key type ._llcrypt registered [ 2057.369760] Key type .llcrypt registered [ 2057.538517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2057.566305] alg: No test for adler32 (adler32-zlib) [ 2063.154300] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:05:15 (1788815115) [ 2063.381530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.407151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.457949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.479941] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2064.058269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.092924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2064.120845] LNet: Removed LNI 192.168.204.25@tcp [ 2064.281139] LNet: 106296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.577404] Key type .llcrypt unregistered [ 2064.579951] Key type ._llcrypt unregistered [ 2073.592530] Key type ._llcrypt registered [ 2073.595518] Key type .llcrypt registered [ 2074.579703] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:05:26 (1788815126) [ 2074.900860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2074.919950] alg: No test for adler32 (adler32-zlib) [ 2075.803909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.860520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2075.892793] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2076.425391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2076.482879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.25@tcp [ 2076.499293] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2076.506290] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2076.510288] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2076, 0, 0 [ 2076.515549] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2076.518517] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2076.522471] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2076.524873] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2076, 0, 0 [ 2076.529750] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2076.532799] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2076.536652] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2076.539706] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2076, 0, 2126 [ 2076.546154] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2076.548680] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2076.553168] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2076.555566] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2076, 0, 2126 [ 2076.559589] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2082.067707] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:05:34 (1788815134) [ 2082.531301] LNet: 107833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.536130] LNetError: 107833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.549870] LNet: Removed LNI 192.168.204.25@tcp [ 2083.219440] Key type .llcrypt unregistered [ 2083.221748] Key type ._llcrypt unregistered [ 2088.215901] Key type ._llcrypt registered [ 2088.218599] Key type .llcrypt registered [ 2088.352797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.366349] alg: No test for adler32 (adler32-zlib) [ 2089.234300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2089.268767] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2095.704617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 2103.859953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.899448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2103.919446] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 2104.752204] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2104.838872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2104.846325] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2104.851630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2107.142422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2107.174490] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2115.728461] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:06:08 (1788815168) [ 2115.806611] LNet: 109328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.814406] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.826522] LNet: Removed LNI 192.168.204.25@tcp [ 2116.831898] LNet: Removed LNI 192.168.204.25@tcp1 [ 2116.842471] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2116.852735] LNetError: 109328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2116.873504] LNet: Removed LNI 10.1.2.3@tcp99 [ 2117.415608] Key type .llcrypt unregistered [ 2117.419147] Key type ._llcrypt unregistered [ 2122.265722] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2122.268115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2124.659893] Key type ._llcrypt registered [ 2124.663031] Key type .llcrypt registered [ 2124.822905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2124.834485] alg: No test for adler32 (adler32-zlib) [ 2125.719814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.772735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2125.793353] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2126.477425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2126.503794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2128.199472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2128.237987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2128.267845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2128.284477] LNetError: 110079:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2128.322163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2128.363649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2128.687425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2128.742972] LNet: 110099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.752322] LNetError: 110099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2128.760398] LNet: Removed LNI 10.1.2.3@tcp [ 2129.829694] LNet: Removed LNI 192.168.204.25@tcp [ 2130.333445] Key type .llcrypt unregistered [ 2130.335753] Key type ._llcrypt unregistered [ 2138.491399] Key type ._llcrypt registered [ 2138.493522] Key type .llcrypt registered [ 2139.396959] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:06:31 (1788815191) [ 2139.767893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.780362] alg: No test for adler32 (adler32-zlib) [ 2140.661145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.727980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.760482] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2141.308067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2141.388338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg425-client@tcp [ 2145.879335] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:06:38 (1788815198) [ 2145.995380] LNet: 111628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.002189] LNetError: 111628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.013820] LNet: Removed LNI 192.168.204.25@tcp [ 2146.482462] Key type .llcrypt unregistered [ 2146.484779] Key type ._llcrypt unregistered [ 2150.855496] Key type ._llcrypt registered [ 2150.858405] Key type .llcrypt registered [ 2150.963222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2150.976283] alg: No test for adler32 (adler32-zlib) [ 2151.823891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2151.846218] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2151.868656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2156.453625] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:06:49 (1788815209) [ 2157.415395] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2158.619857] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:06:51 (1788815211) [ 2159.824394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2160.929853] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:06:53 (1788815213) [ 2161.105131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.141516] LNet: 113048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.146024] LNetError: 113048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.163877] LNet: Removed LNI 192.168.204.25@tcp [ 2161.186488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.238743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.265596] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2161.853876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2161.899990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2166.705115] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:06:59 (1788815219) [ 2166.819799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2166.840440] LNet: 113756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.846535] LNetError: 113756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.863969] LNet: Removed LNI 192.168.204.25@tcp [ 2166.904454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.965463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2167.001257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2167.553480] Key type .llcrypt unregistered [ 2167.556749] Key type ._llcrypt unregistered [ 2176.395530] Key type ._llcrypt registered [ 2176.396654] Key type .llcrypt registered [ 2177.349251] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:07:09 (1788815229) [ 2177.715673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.730452] alg: No test for adler32 (adler32-zlib) [ 2178.603352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.654346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.682729] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2179.401934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2179.453624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2184.616728] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:07:17 (1788815237) [ 2184.702060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.727322] LNet: 115304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.732517] LNetError: 115304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.761945] LNet: Removed LNI 192.168.204.25@tcp [ 2184.789942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.825963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2184.852871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2189.467338] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:07:22 (1788815242) [ 2189.671203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.698209] LNet: 115890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.735983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.803120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2190.092404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2190.126418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2190.174489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2190.507206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2190.537869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2190.607184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.638572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.695502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2190.942225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2190.971203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2191.023254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2191.325553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2191.367120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2196.024267] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:07:28 (1788815248) [ 2196.171293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.190363] LNet: 116523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2196.194898] LNet: 116523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2196.212746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.252842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2196.271446] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2196.748822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2196.770303] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2197.272180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2197.308390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2197.435330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.449565] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2197.459035] LNet: Removed LNI 192.168.204.25@tcp [ 2197.468970] LNet: Removed LNI 192.168.204.25@tcp2 [ 2197.482856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.518401] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2197.522514] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2197.525458] LNet: 116808:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2197.799839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2197.938611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.007186] LNet: Removed LNI 192.168.204.25@tcp [ 2199.013558] LNetError: 116845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.021864] LNetError: 116845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2199.034919] LNet: Removed LNI 192.168.204.25@tcp2 [ 2199.063425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.112632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.129588] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2199.611383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2199.631459] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2200.311456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2200.337825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2200.379979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2200.550409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.567319] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.570953] LNet: 117130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2201.631134] LNet: Removed LNI 192.168.204.25@tcp [ 2201.633764] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.644381] LNetError: 117130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2201.669173] LNet: Removed LNI 192.168.204.25@tcp2 [ 2201.706414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.759854] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2201.764762] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2201.769445] LNet: 117136:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2202.116570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2202.269680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.284981] LNet: Removed LNI 192.168.204.25@tcp [ 2203.305496] LNet: Removed LNI 192.168.204.25@tcp2 [ 2203.334266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.384809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.403069] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2203.953889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2203.975845] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2204.546976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2204.600211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2204.749188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.767402] LNet: 117453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.772429] LNet: 117453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2205.791369] LNet: Removed LNI 192.168.204.25@tcp [ 2205.797519] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.802908] LNetError: 117453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2205.819615] LNet: Removed LNI 192.168.204.25@tcp2 [ 2205.849748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.901432] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2205.909495] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2205.913352] LNet: 117459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.250386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2206.383745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.455196] LNet: Removed LNI 192.168.204.25@tcp [ 2207.463952] LNet: Removed LNI 192.168.204.25@tcp2 [ 2207.490917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.527554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.565404] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2208.047668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2208.061248] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2208.677452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2208.704797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2208.759678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2208.979209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.015692] LNet: Removed LNI 192.168.204.25@tcp [ 2211.045967] LNet: Removed LNI 192.168.204.25@tcp2 [ 2211.071756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.128944] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2211.138864] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2211.144670] LNet: 117787:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2211.580309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2211.785696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2211.815345] LNet: Removed LNI 192.168.204.25@tcp [ 2212.837749] LNet: Removed LNI 192.168.204.25@tcp2 [ 2212.869041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.912197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2212.945561] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2213.614910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2213.635910] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2214.346104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2214.376444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2214.419400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2214.720894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.737643] LNet: 118121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.744749] LNet: 118121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2214.760378] LNetError: 118121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.768096] LNetError: 118121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2214.774605] LNet: Removed LNI 192.168.204.25@tcp [ 2214.794172] LNet: Removed LNI 192.168.204.25@tcp2 [ 2214.834713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.900473] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2214.909794] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2215.313525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2215.582776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.608320] LNet: Removed LNI 192.168.204.25@tcp [ 2216.617836] LNet: Removed LNI 192.168.204.25@tcp2 [ 2216.635374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.687274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.708966] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2217.269838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2217.287217] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2217.923836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2217.944736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2217.968194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2218.018175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2218.301962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.359149] LNet: Removed LNI 192.168.204.25@tcp [ 2220.393435] LNet: Removed LNI 192.168.204.25@tcp2 [ 2220.417752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.477701] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2220.486796] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2220.490549] LNet: 118488:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.501284] LNet: 118488:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2220.867898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2225.766071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2226.790511] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:07:59 (1788815279) [ 2226.897943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.919918] LNet: Removed LNI 192.168.204.25@tcp [ 2227.943262] LNet: Removed LNI 192.168.204.25@tcp2 [ 2227.963768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.179102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.195738] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2228.759738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2228.783072] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.497731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.531662] LNet: Removed LNI 10.1.2.3@tcp [ 2229.543102] LNet: Removed LNI 192.168.204.25@tcp [ 2229.560172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.618805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2229.644165] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2229.648633] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.977025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2229.992388] LNet: Removed LNI 192.168.204.25@tcp [ 2230.002707] LNet: Removed LNI 10.1.2.3@tcp [ 2234.797725] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:08:07 (1788815287) [ 2234.896504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.917428] LNet: 120120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.921011] LNet: 120120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2234.940815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.983701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.999858] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2235.499646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2235.533503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2235.563916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.594940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.626494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.660767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.686534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.713700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.743518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.766902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.797367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.821089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2235.933562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2240.736826] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 17:08:13 (1788815293) [ 2240.914071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.932558] LNetError: 120862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2240.936879] LNetError: 120862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2240.945763] LNet: Removed LNI 192.168.204.25@tcp [ 2240.957768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.001405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.036923] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2241.657811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2241.691884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2241.730785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.761962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.788721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.833483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.859243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.885623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.912609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.946189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2241.996547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.025441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.109723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2242.149518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.177458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.198932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.245194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.277466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.310100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.347997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.379535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.407398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2242.455308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.25@tcp [ 2246.855549] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:08:19 (1788815299) [ 2246.965826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.041135] LNet: Removed LNI 192.168.204.25@tcp [ 2248.067795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.148118] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2248.279954] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2248.283396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2248.286624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2250.484154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.505133] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2251.024477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2251.040782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2252.517180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2252.539917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2252.595525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.25@tcp -m PUT -e local_timeout -r 1 [ 2252.652452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2252.700404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2252.703892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2252.900763] LNetError: 121677:0:(lib-move.c:4794:lnet_parse()) 192.168.204.25@tcp, src 192.168.204.25@tcp: Dropping PUT (error -108 looking up sender) [ 2253.923831] LNet: Removed LNI 10.1.2.3@tcp [ 2254.950075] LNet: Removed LNI 192.168.204.25@tcp [ 2255.229792] Key type .llcrypt unregistered [ 2255.238744] Key type ._llcrypt unregistered [ 2263.019908] Key type ._llcrypt registered [ 2263.021749] Key type .llcrypt registered [ 2263.924656] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:08:36 (1788815316) [ 2264.333894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.345449] alg: No test for adler32 (adler32-zlib) [ 2265.250091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.521882] LNet: 122984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.912782] Key type .llcrypt unregistered [ 2265.917456] Key type ._llcrypt unregistered [ 2269.874553] Key type ._llcrypt registered [ 2269.878493] Key type .llcrypt registered [ 2270.000992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2270.011364] alg: No test for adler32 (adler32-zlib) [ 2270.864738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2270.883607] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2276.845300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 2277.044893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2277.583162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.125@tcp [ 2277.765643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2277.790561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2277.824526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2277.855708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2277.888788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2277.928434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2277.960883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2277.992984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2278.016730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2278.034349] LNet: 123432:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2278.047252] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 2278.052740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2278.082531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2278.118865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2278.147768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2278.181551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2278.213329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2278.740244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2278.915973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2278.938628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2278.974625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2279.002354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2279.029878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2279.055903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2279.077060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2279.111854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2279.136987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2279.151331] LNet: 123551:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2279.172224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2279.211421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2279.243617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2279.281828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2279.316052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2279.343147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2279.870484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.125@tcp [ 2280.033711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2280.073413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2280.101907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2280.128909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2280.154916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2280.196582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2280.224978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2280.247330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2280.270461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2280.291510] LNet: 123671:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2280.312187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2280.343700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2280.372829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2280.407058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2280.436819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2280.485257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2280.976127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2281.145725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2281.171956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2281.189525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2281.212852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2281.241545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2281.276663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2281.296864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2281.323464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2281.365654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2281.405403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2281.439988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2281.468578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2281.499540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2281.554069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2281.593184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2281.848660] LNet: 123836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.855175] LNetError: 123836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.871226] LNet: Removed LNI 192.168.204.25@tcp [ 2282.159382] Key type .llcrypt unregistered [ 2282.164122] Key type ._llcrypt unregistered [ 2293.569100] Key type ._llcrypt registered [ 2293.571242] Key type .llcrypt registered [ 2294.424038] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:09:06 (1788815346) [ 2294.728089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.739269] alg: No test for adler32 (adler32-zlib) [ 2295.615742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.873648] LNet: 124721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.324903] Key type .llcrypt unregistered [ 2296.329608] Key type ._llcrypt unregistered [ 2301.052543] Key type ._llcrypt registered [ 2301.055578] Key type .llcrypt registered [ 2301.248447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2301.258604] alg: No test for adler32 (adler32-zlib) [ 2302.145839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2302.168128] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2308.062994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.125@tcp [ 2314.232252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.262394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2314.279996] LNet: Added LNI 192.168.204.25@tcp1 [8/256/0/180] [ 2314.952495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2315.388543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.125@tcp [ 2315.488753] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.125@tcp [ 2315.575404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.598428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.624155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.652519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.679247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2315.709553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2315.733151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2315.763920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2315.790873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2315.817610] LNet: 125370:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.833571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2315.870907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2315.895986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2315.925636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2315.965277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2316.007691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2316.494843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2316.705367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2316.724420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2316.746956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2316.771288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2316.806093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2316.841989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2316.881814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2316.911237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2316.943732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2316.960533] LNet: 125489:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2316.986675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2317.020956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2317.058141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2317.085548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2317.120485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2317.155579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2317.623255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.125@tcp [ 2317.789931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2317.812578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2317.838775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2317.863200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2317.889124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2317.919482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2317.961977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2317.999184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2318.027309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2318.051385] LNet: 125608:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2318.069183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2318.110251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2318.143164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2318.171349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2318.198170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2318.229245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2318.739525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2318.936548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2318.970334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2318.994328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2319.021479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2319.054230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2319.090019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2319.132411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2319.161600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2319.199800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2319.230318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2319.251857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2319.278077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2319.304315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2319.335578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2319.370881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2323.398525] LNet: 125808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.408703] LNetError: 125808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.418950] LNet: Removed LNI 192.168.204.25@tcp [ 2323.432928] LNet: Removed LNI 192.168.204.25@tcp1 [ 2323.726040] Key type .llcrypt unregistered [ 2323.729870] Key type ._llcrypt unregistered [ 2336.377958] Key type ._llcrypt registered [ 2336.379965] Key type .llcrypt registered [ 2337.267451] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:09:49 (1788815389) [ 2337.592925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.606951] alg: No test for adler32 (adler32-zlib) [ 2338.484312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.519887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.553466] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2339.052678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2339.073854] LNet: Added LNI 192.168.204.25@tcp2 [8/256/0/180] [ 2339.689596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.25@tcp [ 2339.773754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2340.835430] LNet: Removed LNI 192.168.204.25@tcp [ 2345.396578] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:09:57 (1788815397) [ 2346.425251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2347.680711] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:10:00 (1788815400) [ 2352.558161] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:10:05 (1788815405) [ 2358.717845] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:10:10 (1788815410) [ 2364.009242] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:10:16 (1788815416) [ 2369.590660] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:10:21 (1788815421) [ 2375.255515] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:10:27 (1788815427) [ 2381.151602] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:10:33 (1788815433) [ 2386.016245] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:10:38 (1788815438) [ 2390.566788] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:10:43 (1788815443) [ 2394.992196] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:10:47 (1788815447) [ 2399.888835] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:10:52 (1788815452) [ 2405.129321] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:10:57 (1788815457) [ 2409.905474] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:11:02 (1788815462) [ 2414.886473] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:11:07 (1788815467) [ 2420.338881] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:11:12 (1788815472) [ 2425.618967] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:11:18 (1788815478) [ 2430.816959] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:11:23 (1788815483) [ 2435.966717] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:11:28 (1788815488) [ 2436.042590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2436.068157] LNet: 137483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.076766] LNetError: 137483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2436.089186] LNet: Removed LNI 192.168.204.25@tcp2 [ 2436.109273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2436.200385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2436.216596] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2437.046883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.065613] LNet: 137633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.072437] LNetError: 137633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.085328] LNet: Removed LNI 192.168.204.25@tcp [ 2437.102638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.141709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2437.163536] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2441.516913] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:11:34 (1788815494) [ 2446.181224] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:11:38 (1788815498) [ 2451.139084] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:11:43 (1788815503) [ 2456.145921] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:11:48 (1788815508) [ 2460.923325] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:11:53 (1788815513) [ 2465.758280] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:11:58 (1788815518) [ 2470.071327] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:12:02 (1788815522) [ 2474.465456] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:12:07 (1788815527) [ 2478.970136] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:12:11 (1788815531) [ 2484.290744] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:12:16 (1788815536) [ 2489.194798] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:12:21 (1788815541) [ 2493.950519] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:12:26 (1788815546) [ 2498.824832] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2173 sec ========== 17:12:31 (1788815551) [ 2499.090633] LNet: 145095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.095340] LNetError: 145095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.110066] LNet: Removed LNI 192.168.204.25@tcp [ 2499.580434] Key type .llcrypt unregistered [ 2499.583974] Key type ._llcrypt unregistered [ 2508.712133] Key type ._llcrypt registered [ 2508.716267] Key type .llcrypt registered [ 2508.895335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2508.904280] alg: No test for adler32 (adler32-zlib) [ 2509.911365] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2510.089905] LNet: Added LNI 192.168.204.25@tcp [8/256/0/180] [ 2511.735323] Key type lgssc registered [ 2512.657645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.197312] Lustre: Mounted lustre-client - version 2.17.58_40_gea5fd3f [ 2559.500978] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2564.310477] Lustre: Unmounted lustre-client [ 2579.547740] Key type lgssc unregistered [ 2579.754851] LNet: 148037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2579.759335] LNetError: 148037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2579.773311] LNet: Removed LNI 192.168.204.25@tcp [ 2580.278142] Key type .llcrypt unregistered [ 2580.280784] Key type ._llcrypt unregistered