[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 453957176 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.003182] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005022] kvm-guest: setup PV IPIs [ 0.008581] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011023] pid_max: default: 32768 minimum: 301 [ 0.012185] LSM: Security Framework initializing [ 0.013094] Yama: becoming mindful. [ 0.014054] SELinux: Initializing. [ 0.016072] *** VALIDATE selinux *** [ 0.024451] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029090] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032151] *** VALIDATE tmpfs *** [ 0.034023] *** VALIDATE proc *** [ 0.035281] *** VALIDATE cgroup *** [ 0.036013] *** VALIDATE cgroup2 *** [ 0.037291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040036] Spectre V2 : User space: Vulnerable [ 0.041014] Speculative Store Bypass: Vulnerable [ 0.044577] debug: unmapping init [mem 0xffffffffaf459000-0xffffffffaf460fff] [ 0.046919] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047735] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048042] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053018] ... fixed-purpose events: 3 [ 0.054020] ... event mask: 000000070000000f [ 0.055375] rcu: Hierarchical SRCU implementation. [ 0.057784] smp: Bringing up secondary CPUs ... [ 0.058622] x86: Booting SMP configuration: [ 0.059042] .... node #0, CPUs: #1 #2 #3 [ 0.063112] smp: Brought up 1 node, 4 CPUs [ 0.065020] smpboot: Max logical packages: 1 [ 0.066016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147658] node 0 deferred pages initialised in 79ms [ 0.150267] devtmpfs: initialized [ 0.152437] x86/mm: Memory block size: 128MB [ 0.156227] gcov: version magic: 0x41383552 [ 0.158364] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162156] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165382] pinctrl core: initialized pinctrl subsystem [ 0.167258] [ 0.167854] ************************************************************* [ 0.170022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173022] ** ** [ 0.175020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178025] ** ** [ 0.181021] ** This means that this kernel is built to expose internal ** [ 0.183024] ** IOMMU data structures, which may compromise security on ** [ 0.186019] ** your system. ** [ 0.189034] ** ** [ 0.192022] ** If you see this message and you are not debugging the ** [ 0.195020] ** kernel, report this immediately to your vendor! ** [ 0.197024] ** ** [ 0.201020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203021] ************************************************************* [ 0.207641] NET: Registered protocol family 16 [ 0.209638] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.215086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.219149] cpuidle: using governor menu [ 0.220991] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221621] PCI: Using configuration type 1 for base access [ 0.222159] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237088] cryptd: max_cpu_qlen set to 1000 [ 0.239311] ACPI: Added _OSI(Module Device) [ 0.242024] ACPI: Added _OSI(Processor Device) [ 0.243018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250264] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258394] ACPI: Interpreter enabled [ 0.260092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.262017] ACPI: Using IOAPIC for interrupt routing [ 0.264231] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.268511] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.279063] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.281051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.284030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293568] acpiphp: Slot [2] registered [ 0.295197] acpiphp: Slot [5] registered [ 0.297202] acpiphp: Slot [6] registered [ 0.298281] acpiphp: Slot [3] registered [ 0.300162] acpiphp: Slot [4] registered [ 0.302142] acpiphp: Slot [7] registered [ 0.304150] acpiphp: Slot [8] registered [ 0.305142] acpiphp: Slot [9] registered [ 0.307133] acpiphp: Slot [10] registered [ 0.309159] acpiphp: Slot [11] registered [ 0.310191] acpiphp: Slot [12] registered [ 0.312152] acpiphp: Slot [13] registered [ 0.315191] acpiphp: Slot [14] registered [ 0.317154] acpiphp: Slot [15] registered [ 0.319152] acpiphp: Slot [16] registered [ 0.320146] acpiphp: Slot [17] registered [ 0.322159] acpiphp: Slot [18] registered [ 0.323116] acpiphp: Slot [19] registered [ 0.325142] acpiphp: Slot [20] registered [ 0.326148] acpiphp: Slot [21] registered [ 0.328147] acpiphp: Slot [22] registered [ 0.329176] acpiphp: Slot [23] registered [ 0.331125] acpiphp: Slot [24] registered [ 0.332133] acpiphp: Slot [25] registered [ 0.334144] acpiphp: Slot [26] registered [ 0.335144] acpiphp: Slot [27] registered [ 0.337143] acpiphp: Slot [28] registered [ 0.339134] acpiphp: Slot [29] registered [ 0.341171] acpiphp: Slot [30] registered [ 0.342165] acpiphp: Slot [31] registered [ 0.344135] PCI host bridge to bus 0000:00 [ 0.346030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.348031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351083] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.356044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359065] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361227] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365105] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.368633] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.377809] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.383068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.386030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.389037] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394603] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.397855] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.400053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402825] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.415966] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.428337] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.441021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.456022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.471428] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.504024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.514715] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.517503] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.520463] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.523517] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.526317] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.532260] iommu: Default domain type: Passthrough [ 0.534527] SCSI subsystem initialized [ 0.536207] ACPI: bus type USB registered [ 0.537151] usbcore: registered new interface driver usbfs [ 0.539108] usbcore: registered new interface driver hub [ 0.543111] usbcore: registered new device driver usb [ 0.545260] pps_core: LinuxPPS API ver. 1 registered [ 0.547017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.550089] PTP clock support registered [ 0.552188] EDAC MC: Ver: 3.0.0 [ 0.555054] PCI: Using ACPI for IRQ routing [ 0.556984] NetLabel: Initializing [ 0.559021] NetLabel: domain hash size = 128 [ 0.560015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.563114] NetLabel: unlabeled traffic allowed by default [ 0.566162] vgaarb: loaded [ 0.568214] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.569016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.575345] clocksource: Switched to clocksource kvm-clock [ 0.688933] VFS: Disk quotas dquot_6.6.0 [ 0.690776] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.693765] *** VALIDATE ramfs *** [ 0.695261] *** VALIDATE hugetlbfs *** [ 0.697117] pnp: PnP ACPI init [ 0.699794] pnp: PnP ACPI: found 6 devices [ 0.720749] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.724618] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.726980] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.729629] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.732521] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.735397] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.738846] NET: Registered protocol family 2 [ 0.741755] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.747415] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.751674] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.757150] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.760626] TCP: Hash tables configured (established 65536 bind 65536) [ 0.764184] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.768161] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.771476] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.774933] NET: Registered protocol family 1 [ 0.778041] RPC: Registered named UNIX socket transport module. [ 0.780548] RPC: Registered udp transport module. [ 0.782186] RPC: Registered tcp transport module. [ 0.783868] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.786849] NET: Registered protocol family 44 [ 0.789060] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.791519] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.793460] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.795514] PCI: CLS 0 bytes, default 64 [ 0.797444] Unpacking initramfs... [ 2.255447] debug: unmapping init [mem 0xffff9f23bcc64000-0xffff9f23bffcffff] [ 2.262556] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.265501] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.270104] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.802826] Initialise system trusted keyrings [ 2.807255] Key type blacklist registered [ 2.810304] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.819741] zbud: loaded [ 2.822988] *** VALIDATE nfs *** [ 2.824193] *** VALIDATE nfs4 *** [ 2.826216] pstore: using deflate compression [ 2.831793] Platform Keyring initialized [ 2.956167] NET: Registered protocol family 38 [ 2.958912] Key type asymmetric registered [ 2.960707] Asymmetric key parser 'x509' registered [ 2.962857] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.967178] io scheduler mq-deadline registered [ 2.968676] io scheduler kyber registered [ 2.970109] io scheduler bfq registered [ 2.972236] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.975346] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.978598] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.981655] ACPI: Power Button [PWRF] [ 2.988768] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.996519] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.006792] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.037119] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.064992] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.070267] Non-volatile memory driver v1.3 [ 3.072026] Linux agpgart interface v0.103 [ 3.110413] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.114558] vda: detected capacity change from 0 to 77115392 [ 3.136629] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.140307] vdb: detected capacity change from 0 to 1073741824 [ 3.149894] libphy: Fixed MDIO Bus: probed [ 3.156487] usbcore: registered new interface driver usbserial_generic [ 3.158862] usbserial: USB Serial support registered for generic [ 3.161249] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.165616] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.167862] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.170667] mousedev: PS/2 mouse device common for all mice [ 3.173864] rtc_cmos 00:05: RTC can wake from S4 [ 3.177242] rtc_cmos 00:05: registered as rtc0 [ 3.178771] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.179274] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.186654] intel_pstate: CPU model not supported [ 3.189545] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.194358] hid: raw HID events driver (C) Jiri Kosina [ 3.196605] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.197366] usbcore: registered new interface driver usbhid [ 3.203670] usbhid: USB HID core driver [ 3.205567] drop_monitor: Initializing network drop monitor service [ 3.208454] Initializing XFRM netlink socket [ 3.210797] NET: Registered protocol family 10 [ 3.213902] Segment Routing with IPv6 [ 3.215457] NET: Registered protocol family 17 [ 3.217526] mpls_gso: MPLS GSO support [ 3.223881] RAS: Correctable Errors collector initialized. [ 3.225922] AVX version of gcm_enc/dec engaged. [ 3.227655] AES CTR mode by8 optimization enabled [ 3.318510] sched_clock: Marking stable (3318487260, 0)->(4320360097, -1001872837) [ 3.322300] registered taskstats version 1 [ 3.325245] Loading compiled-in X.509 certificates [ 3.327095] zswap: loaded using pool lzo/zbud [ 3.365197] Key type big_key registered [ 3.379588] Key type encrypted registered [ 3.381586] ima: No TPM chip found, activating TPM-bypass! [ 3.384059] ima: Allocated hash algorithm: sha1 [ 3.385984] ima: No architecture policies found [ 3.388275] evm: Initialising EVM extended attributes: [ 3.390331] evm: security.selinux [ 3.391596] evm: security.ima [ 3.392833] evm: security.capability [ 3.395061] evm: HMAC attrs: 0x1 [ 3.400260] rtc_cmos 00:05: setting system clock to 2026-10-02 17:59:15 UTC (1790963955) [ 3.407058] debug: unmapping init [mem 0xffffffffb0403000-0xffffffffb05fffff] [ 3.410668] debug: unmapping init [mem 0xffffffffaf182000-0xffffffffaf458fff] [ 3.420204] Write protecting the kernel read-only data: 28672k [ 3.424113] debug: unmapping init [mem 0xffffffffad803000-0xffffffffad9fffff] [ 3.427319] debug: unmapping init [mem 0xffffffffae114000-0xffffffffae1fffff] [ 3.470607] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.480509] systemd[1]: Detected virtualization kvm. [ 3.482771] systemd[1]: Detected architecture x86-64. [ 3.484880] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.512438] systemd[1]: No hostname configured. [ 3.514317] systemd[1]: Set hostname to . [ 3.516307] random: systemd: uninitialized urandom read (16 bytes read) [ 3.518995] systemd[1]: Initializing machine ID from random generator. [ 3.572729] random: ln: uninitialized urandom read (6 bytes read) [ 3.681163] random: systemd: uninitialized urandom read (16 bytes read) [ 3.684594] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.689890] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.695180] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.399992] device-mapper: uevent: version 1.0.3 [ 4.403716] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.236169] virtio_net virtio0 ens2: renamed from eth0 [ 5.350056] scsi host0: ata_piix [ 5.386743] scsi host1: ata_piix [ 5.389043] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.392091] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.795829] dracut-initqueue[600]: RTNETLINK answers: File exists [ 9.903376] random: crng init done [ 9.905041] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.491518] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.776475] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.066171] SELinux: Disabled at runtime. [ 12.132615] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.142959] systemd[1]: Detected virtualization kvm. [ 12.145114] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.729547] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.732592] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.737281] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.740124] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.743177] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.751055] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.768764] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 12.926364] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.401370] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.672282] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.730700] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.872195] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.897346] EDAC sbridge: Ver: 1.1.2 [ 15.570896] Key type dns_resolver registered [ 15.908234] NFS: Registering the id_resolver key type [ 15.910509] Key type id_resolver registered [ 15.912360] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg443-client login: [ 44.782280] hrtimer: interrupt took 3272997 ns [ 67.764146] libcfs: loading out-of-tree module taints kernel. [ 67.819667] Key type ._llcrypt registered [ 67.824137] Key type .llcrypt registered [ 68.286949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 68.309620] alg: No test for adler32 (adler32-zlib) [ 70.013702] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 70.823073] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 72.689124] Key type lgssc registered [ 75.205169] Lustre: Echo OBD driver; http://www.lustre.org/ [ 222.369388] Lustre: Mounted lustre-client - version 2.17.59_2_g3d9f037 [ 227.934835] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.667404] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing check_logdir /tmp/testlogs/ [ 246.720408] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing yml_node [ 248.295044] Lustre: lustre-OST0000-osc-ffff9f24206f5000: disconnect after 23s idle [ 251.523412] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 254.482553] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 257.618936] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 259.398258] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 2 14:03:29 EDT 2026 [ 275.602029] Lustre: Unmounted lustre-client [ 316.061100] Key type lgssc unregistered [ 316.402150] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.411374] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.458934] LNet: Removed LNI 192.168.204.43@tcp [ 317.545252] Key type .llcrypt unregistered [ 317.546916] Key type ._llcrypt unregistered [ 343.228688] Key type ._llcrypt registered [ 343.233076] Key type .llcrypt registered [ 343.436170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 343.453022] alg: No test for adler32 (adler32-zlib) [ 344.410967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 344.495391] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 344.536399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 348.643564] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 348.858366] LNet: 9316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.862484] LNetError: 9316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.887493] LNet: Removed LNI 192.168.204.43@tcp [ 349.743334] Key type .llcrypt unregistered [ 349.748491] Key type ._llcrypt unregistered [ 356.285692] Key type ._llcrypt registered [ 356.287472] Key type .llcrypt registered [ 357.441200] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:05:08 (1790964308) [ 357.945124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 357.956298] alg: No test for adler32 (adler32-zlib) [ 358.889611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.966531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.985894] LNet: 9726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.001283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.045579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 359.070860] LNet: 9733:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 366.775019] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:05:17 (1790964317) [ 366.894881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.935725] LNet: 10304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.981693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.057089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 373.675288] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:05:24 (1790964324) [ 373.846850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.909756] LNet: 10897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.933835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.000406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 379.880450] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:05:30 (1790964330) [ 380.182823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.232632] LNet: 11482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.269618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.377391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 387.653068] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:05:38 (1790964338) [ 387.726431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.751302] LNet: 12058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.818714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.013744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 388.590628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 396.708307] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:05:47 (1790964347) [ 396.950707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.992264] LNet: 12656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.048643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.194646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 397.662836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 398.165611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 404.589296] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:05:55 (1790964355) [ 405.005753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.075683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.175572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 412.488433] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:06:03 (1790964363) [ 412.586179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.649476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.726776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 412.758340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 413.240235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 413.307925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 414.033530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 414.070490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 414.707651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 414.750122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 415.167721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 415.231082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 415.670410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 415.710286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 416.222262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 416.276126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 416.847796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 416.876967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 417.429595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 417.463286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 417.945604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 417.991538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 418.551169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 418.593840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 419.210648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 419.241659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 419.636732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 419.674819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 425.875696] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:06:16 (1790964376) [ 425.974038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.988387] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.998252] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 426.026219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.135179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 426.165770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 432.083483] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:06:23 (1790964383) [ 432.234546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.285164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.366094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 432.401283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 438.284985] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:06:29 (1790964389) [ 438.391984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.421093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.493757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 438.531468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 444.499759] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:06:35 (1790964395) [ 444.636529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.699800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.787303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 444.838295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 451.085526] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:06:41 (1790964401) [ 451.225796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.276638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.350768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 451.387130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 457.096902] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:06:48 (1790964408) [ 457.272868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.321974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.385439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 457.417520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 462.672368] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:06:53 (1790964413) [ 462.772956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.807634] LNet: 18032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.812271] LNet: 18032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 462.829978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.930886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.337158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.259492] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:07:00 (1790964420) [ 469.343940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.393584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.600931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.812075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.768953] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:07:06 (1790964426) [ 476.028197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.088415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.215871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.414591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.810948] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:07:13 (1790964433) [ 482.913077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.946823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.486652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.752410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.782647] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:07:20 (1790964440) [ 489.910740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.950217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.052266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.123197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.343494] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:07:26 (1790964446) [ 496.488879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.610437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.794083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.863154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.424681] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:07:34 (1790964454) [ 504.521452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.597043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.671799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 511.948738] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:07:42 (1790964462) [ 512.037307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.087057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.178628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 520.243488] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:07:50 (1790964470) [ 520.343734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.393514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.541533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 521.245679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 527.522086] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:07:58 (1790964478) [ 527.629798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.675441] LNet: 23476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.680912] LNet: 23476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 527.717862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.830908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 527.859388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 534.553977] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:08:05 (1790964485) [ 534.752756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.822829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.937572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 534.991703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 541.207063] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:08:12 (1790964492) [ 541.297128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.354197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.467719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 541.523453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 548.952080] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:08:19 (1790964499) [ 549.147463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.228291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.288849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 549.333203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 557.563968] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:08:28 (1790964508) [ 557.668259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.754932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.879144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 557.971103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 557.996641] LNetError: 25837:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 558.060930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 558.121180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 566.568983] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:08:36 (1790964516) [ 566.717931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.779888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.933484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gHukTL1tFT [ 567.005928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gHukTL1tFT [ 567.053808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.gHukTL1tFT [ 573.697151] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:08:44 (1790964524) [ 573.888230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.959060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.056224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 574.090323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 574.147354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 580.299346] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:08:51 (1790964531) [ 580.384112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.421051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.528072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 580.561326] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 582.201995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 583.696900] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 585.592229] Key type lgssc registered [ 586.697663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 589.580705] LNetError: 27595:0:(lib-move.c:2173:lnet_handle_find_routed_path()) peer 192.168.204.43@tcp99 has no available nets [ 591.747363] Key type lgssc unregistered [ 592.095649] LNetError: 28714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.112178] LNet: Removed LNI 192.168.204.43@tcp [ 592.623582] Key type .llcrypt unregistered [ 592.627746] Key type ._llcrypt unregistered [ 598.593258] Key type ._llcrypt registered [ 598.597703] Key type .llcrypt registered [ 599.699141] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:09:10 (1790964550) [ 600.312432] Key type .llcrypt unregistered [ 600.323933] Key type ._llcrypt unregistered [ 606.038922] Key type ._llcrypt registered [ 606.041099] Key type .llcrypt registered [ 606.319392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.340783] alg: No test for adler32 (adler32-zlib) [ 607.300546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.375618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 607.413775] LNet: 29636:0:(router.c:1482:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 607.419394] LNet: 29636:0:(router.c:1504:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 607.425057] LNet: 29636:0:(router.c:1526:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 607.447333] LNet: Message forwarding will be enabled in 2s [ 610.528149] LNet: Message forwarding enabled [ 610.906209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 610.945628] LNet: Message forwarding will stop in 2s to 6s [ 614.628199] LNet: Message forwarding disabled [ 615.568553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.615388] LNet: 29734:0:(router.c:1482:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 615.628134] LNet: 29734:0:(router.c:1504:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 615.633753] LNet: 29734:0:(router.c:1526:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 615.645522] LNet: Message forwarding will be enabled in 2s [ 615.752977] LNet: 29747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.207549] Key type .llcrypt unregistered [ 616.214950] Key type ._llcrypt unregistered [ 627.161143] Key type ._llcrypt registered [ 627.163236] Key type .llcrypt registered [ 628.401662] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:09:39 (1790964579) [ 628.719327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.744841] alg: No test for adler32 (adler32-zlib) [ 629.650453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.727469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 635.265817] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:09:46 (1790964586) [ 635.378188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.408550] LNet: 31148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.437374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.497065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 635.535329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 635.577195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 635.615276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 635.648650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 635.687197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 635.709850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 635.734927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 635.762214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 635.789250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 635.811463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 635.853517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 635.916776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 635.954268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 635.994803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 636.023854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 636.078476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 636.118941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 636.155737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 636.174362] LNetError: 31192:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 641.597138] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:09:52 (1790964592) [ 641.856196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.894066] LNet: 31763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.921267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.992654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.122534] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:09:59 (1790964599) [ 650.167263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg443-server [ 651.947629] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:10:02 (1790964602) [ 652.226899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.250188] LNet: 32693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.272758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.351205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 652.377965] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 653.354544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[44-48]@tcp [ 653.374789] LNet: 32826:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 653.423089] LNetError: 32697:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 653.446982] LNetError: 32697:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.44@tcp has gone from up to down [ 660.423980] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:10:11 (1790964611) [ 662.047540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg443-server [ 663.495287] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:10:14 (1790964614) [ 663.692424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.705785] LNetError: 32697:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.47@tcp has gone from up to down [ 663.711766] LNetError: 32697:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 663.716044] LNet: 33763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.719627] LNetError: 33763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.740464] LNet: Removed LNI 192.168.204.43@tcp [ 663.763514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.840806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.871434] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 664.434433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 664.453587] LNet: 33880:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 664.466538] LNet: 33880:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 664.488165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[44-49/2]@tcp [ 664.512463] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.48@tcp is being used as a gateway but routing feature is not turned on [ 664.524744] LNetError: 33767:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 670.683520] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:10:21 (1790964621) [ 671.100345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.129060] LNet: 34474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.140298] LNetError: 34474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.168524] LNet: Removed LNI 192.168.204.43@tcp [ 671.190905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.282738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 671.344306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.376328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.448505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.509693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.558463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.645639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 671.745903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.788549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.856472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.926595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.966987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.027789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 672.116393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.168302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.236547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.327708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.356786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.402643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 672.523166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.590375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.677615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.771601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.817919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.895844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 673.004848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.043824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.121099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 673.233676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.278680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.342328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 673.390508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.427367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.489714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.828286] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:10:30 (1790964630) [ 681.552931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg443-server [ 683.486852] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:10:34 (1790964634) [ 685.628919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg443-server [ 687.625579] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:10:38 (1790964638) [ 688.067488] LNet: 35874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.081833] LNet: 35874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 688.683407] Key type .llcrypt unregistered [ 688.685815] Key type ._llcrypt unregistered [ 694.833040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 695.112163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 695.124455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 695.139577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 697.767171] Key type ._llcrypt registered [ 697.769535] Key type .llcrypt registered [ 697.964875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.990310] alg: No test for adler32 (adler32-zlib) [ 698.995834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.052647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.091220] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 700.125880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 700.161539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 702.611678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 702.665822] LNet: Removed LNI 10.1.2.3@tcp [ 702.882266] LNet: 36594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.896579] LNetError: 36594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.920261] LNet: Removed LNI 192.168.204.43@tcp [ 703.834994] Key type .llcrypt unregistered [ 703.838962] Key type ._llcrypt unregistered [ 719.206311] Key type ._llcrypt registered [ 719.212879] Key type .llcrypt registered [ 720.647773] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:11:11 (1790964671) [ 721.544339] Key type .llcrypt unregistered [ 721.558877] Key type ._llcrypt unregistered [ 731.167213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.313436] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.316264] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.320106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 734.216888] Key type ._llcrypt registered [ 734.218574] Key type .llcrypt registered [ 734.520981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 734.546516] alg: No test for adler32 (adler32-zlib) [ 735.563170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.650803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.700379] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 736.736638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 736.755479] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.828505] LNet: 38086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.834719] LNetError: 38086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 738.838863] LNet: Removed LNI 10.1.2.3@tcp [ 738.847498] LNet: Removed LNI 192.168.204.43@tcp [ 739.379746] Key type .llcrypt unregistered [ 739.382314] Key type ._llcrypt unregistered [ 755.146647] Key type ._llcrypt registered [ 755.148745] Key type .llcrypt registered [ 756.807741] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:11:47 (1790964707) [ 757.423975] Key type .llcrypt unregistered [ 757.426064] Key type ._llcrypt unregistered [ 764.572544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.717406] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 764.721886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.732200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.480500] Key type ._llcrypt registered [ 767.482748] Key type .llcrypt registered [ 767.747777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.774191] alg: No test for adler32 (adler32-zlib) [ 768.679576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.739784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 768.760899] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 771.075928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 771.119429] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.251639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 773.272954] LNet: Removed LNI 10.1.2.3@tcp [ 773.299759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 773.339749] LNet: Removed LNI 10.1.2.31@tcp [ 773.357884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 773.391721] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.091942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 774.120047] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 775.731925] LNet: 40001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.743450] LNetError: 40001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 775.754729] LNet: Removed LNI 10.1.2.31@tcp [ 775.769827] LNet: Removed LNI 10.1.2.3@tcp [ 776.921517] Key type .llcrypt unregistered [ 776.924611] Key type ._llcrypt unregistered [ 796.676704] Key type ._llcrypt registered [ 796.685342] Key type .llcrypt registered [ 798.336219] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:12:28 (1790964748) [ 798.667887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 798.690236] alg: No test for adler32 (adler32-zlib) [ 799.775272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.817725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.833543] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 800.857509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.927413] LNet: 40955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.939674] LNetError: 40955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.966360] LNet: Removed LNI 192.168.204.43@tcp [ 801.015571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.117929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.43@tcp [ 801.199760] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 802.188843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.43@tcp [ 802.232289] LNet: Removed LNI 192.168.204.43@tcp [ 802.319734] LNet: 41090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.326772] Key type .llcrypt unregistered [ 803.335118] Key type ._llcrypt unregistered [ 818.171154] Key type ._llcrypt registered [ 818.175226] Key type .llcrypt registered [ 819.215320] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:12:50 (1790964770) [ 821.576777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg443-server [ 823.311765] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:12:54 (1790964774) [ 823.808542] Key type .llcrypt unregistered [ 823.812889] Key type ._llcrypt unregistered [ 830.669633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 830.861560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 830.867196] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 830.872079] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 833.545891] Key type ._llcrypt registered [ 833.547403] Key type .llcrypt registered [ 833.732059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.745259] alg: No test for adler32 (adler32-zlib) [ 834.740514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.807844] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 835.532844] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 837.991484] LNet: 42925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.000801] LNetError: 42925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.007625] LNet: Removed LNI 192.168.204.43@tcp1 [ 838.021785] LNet: Removed LNI 10.1.2.3@tcp2 [ 838.665537] Key type .llcrypt unregistered [ 838.671905] Key type ._llcrypt unregistered [ 850.022988] Key type ._llcrypt registered [ 850.025976] Key type .llcrypt registered [ 851.588964] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:13:21 (1790964801) [ 852.341822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 852.356790] alg: No test for adler32 (adler32-zlib) [ 853.341731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.425540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.464365] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 854.402429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.423269] LNet: 43894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.429422] LNetError: 43894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.445216] LNet: Removed LNI 192.168.204.43@tcp [ 854.488469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.586982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 854.615577] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 862.029803] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:13:32 (1790964812) [ 862.226798] LNet: 44486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.234163] LNetError: 44486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.254291] LNet: Removed LNI 192.168.204.43@tcp [ 862.881819] Key type .llcrypt unregistered [ 862.886271] Key type ._llcrypt unregistered [ 871.151598] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.633172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.643964] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.649423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 874.658131] Key type ._llcrypt registered [ 874.659615] Key type .llcrypt registered [ 875.056506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.081782] alg: No test for adler32 (adler32-zlib) [ 876.156789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.273359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.350270] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 877.417717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 877.434872] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 880.603469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.641551] LNet: 45227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.649412] LNetError: 45227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.659977] LNet: Removed LNI 10.1.2.3@tcp [ 880.682647] LNet: Removed LNI 192.168.204.43@tcp [ 880.720567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.803541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 880.844531] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 880.860107] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.713459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.758454] LNet: 45254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.780201] LNetError: 45254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.791974] LNetError: 45254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 881.804239] LNet: Removed LNI 10.1.2.3@tcp [ 881.827023] LNet: Removed LNI 192.168.204.43@tcp [ 881.859552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.965984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 882.012061] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 882.023814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.693101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 882.733098] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 883.819926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.862374] LNet: 45417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.870288] LNetError: 45417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.874572] LNetError: 45417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.879878] LNet: Removed LNI 10.1.2.3@tcp [ 883.882289] LNet: Removed LNI 192.168.204.43@tcp [ 883.894605] LNet: Removed LNI 192.168.204.43@tcp2 [ 883.921992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.012608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.049833] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 884.053538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.060526] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 892.781737] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:14:03 (1790964843) [ 893.408849] LNet: 46014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.419846] LNetError: 46014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.428205] LNetError: 46014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 893.432371] LNet: Removed LNI 10.1.2.3@tcp [ 893.436996] LNet: Removed LNI 192.168.204.43@tcp [ 893.450046] LNet: Removed LNI 192.168.204.43@tcp2 [ 894.194189] Key type .llcrypt unregistered [ 894.196986] Key type ._llcrypt unregistered [ 904.017903] Key type ._llcrypt registered [ 904.027746] Key type .llcrypt registered [ 904.705714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.782464] alg: No test for adler32 (adler32-zlib) [ 905.957854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.136906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.171248] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 907.967562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.026350] LNet: 46409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.035846] LNetError: 46409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.051704] LNet: Removed LNI 192.168.204.43@tcp [ 908.095938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.248077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.303908] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 909.209665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.257408] LNet: 46440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.262609] LNetError: 46440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.279993] LNet: Removed LNI 192.168.204.43@tcp [ 909.315899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.427374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.456336] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 909.789930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.825199] LNet: Removed LNI 192.168.204.43@tcp [ 909.859692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.912548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.952993] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 910.424649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.048287] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.067831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.080981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.772611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.793213] LNet: 46550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.800402] LNet: 46550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 913.812655] LNetError: 46550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.820959] LNetError: 46550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 913.845202] LNet: Removed LNI 192.168.204.43@tcp [ 913.907482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.017216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 914.049776] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 928.140890] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:14:38 (1790964878) [ 928.384603] LNet: 47140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.442377] LNetError: 47140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.499921] LNet: Removed LNI 192.168.204.43@tcp [ 929.827511] Key type .llcrypt unregistered [ 929.829231] Key type ._llcrypt unregistered [ 938.358560] Key type ._llcrypt registered [ 938.371937] Key type .llcrypt registered [ 938.814416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.841861] alg: No test for adler32 (adler32-zlib) [ 939.805719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.864625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.885475] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 940.993448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.030386] LNet: 47547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.040882] LNetError: 47547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.060308] LNet: Removed LNI 192.168.204.43@tcp [ 941.085791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.180727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 941.224601] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 950.548282] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:15:01 (1790964901) [ 950.762870] LNet: 48138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.775648] LNetError: 48138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.841996] LNet: Removed LNI 192.168.204.43@tcp [ 951.391995] Key type .llcrypt unregistered [ 951.394952] Key type ._llcrypt unregistered [ 958.711434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.995053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 959.000958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 959.007169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.611874] Key type ._llcrypt registered [ 961.614184] Key type .llcrypt registered [ 961.880917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.925392] alg: No test for adler32 (adler32-zlib) [ 962.854507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.901870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.921719] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 963.590571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.613698] LNet: 48589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.620796] LNetError: 48589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.636374] LNet: Removed LNI 192.168.204.43@tcp [ 963.657308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.729813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 963.763041] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 971.302501] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:15:21 (1790964921) [ 971.500896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.519696] LNet: 49187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.524765] LNetError: 49187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.546995] LNet: Removed LNI 192.168.204.43@tcp [ 971.585502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.681345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 979.214951] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:15:29 (1790964929) [ 979.471297] LNet: 49769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.180613] Key type .llcrypt unregistered [ 980.183390] Key type ._llcrypt unregistered [ 986.752890] Key type ._llcrypt registered [ 986.758170] Key type .llcrypt registered [ 987.040570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.069259] alg: No test for adler32 (adler32-zlib) [ 988.062627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.120447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.153344] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 989.443846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.481864] LNet: 50165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.496426] LNetError: 50165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.537688] LNet: Removed LNI 192.168.204.43@tcp [ 989.559556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.681438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 989.715720] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 998.199035] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:15:48 (1790964948) [ 998.364766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.421951] LNet: 50751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.446275] LNetError: 50751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.519304] LNet: Removed LNI 192.168.204.43@tcp [ 998.576077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.728784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1006.965449] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:15:57 (1790964957) [ 1009.277634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1010.882152] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:16:01 (1790964961) [ 1012.931717] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1015.081525] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:16:05 (1790964965) [ 1017.540177] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1019.556905] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:16:10 (1790964970) [ 1021.063850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1023.344360] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:16:13 (1790964973) [ 1025.287157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1027.723262] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:16:17 (1790964977) [ 1027.828581] LNet: 52775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.726076] Key type .llcrypt unregistered [ 1028.730564] Key type ._llcrypt unregistered [ 1037.077353] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1037.210367] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1037.220574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1037.225326] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.024295] Key type ._llcrypt registered [ 1040.028867] Key type .llcrypt registered [ 1040.312280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.375586] alg: No test for adler32 (adler32-zlib) [ 1041.330664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.468839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1041.509540] LNet: Added LNI 192.168.204.43@tcp [8/512/0/180] [ 1041.515902] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1041.520165] LNet: 53107:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1041.785649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1041.864413] LNet: 53126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.910057] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.926822] LNet: Removed LNI 10.1.2.3@tcp [ 1041.935418] LNetError: 53126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.004707] LNet: Removed LNI 192.168.204.43@tcp [ 1042.048532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.117852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.137794] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1042.216406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.238814] LNet: 53150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.247037] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.265604] LNet: Removed LNI 192.168.204.43@tcp [ 1042.296814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.386818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.440231] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1042.459635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.464090] LNet: 53158:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1042.644504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.706151] LNet: 53176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.737095] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.745329] LNet: Removed LNI 10.1.2.3@tcp [ 1042.747272] LNetError: 53176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.769048] LNet: Removed LNI 192.168.204.43@tcp [ 1042.802530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.927297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1042.982406] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1042.990066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.997938] LNet: 53184:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1043.294036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.320858] LNet: 53210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.327151] LNetError: 53210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.335613] LNet: Removed LNI 10.1.2.3@tcp [ 1043.338727] LNetError: 53210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.359353] LNet: Removed LNI 192.168.204.43@tcp [ 1043.391411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.445907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.508215] LNet: Added LNI 192.168.204.43@tcp [8/256/0/182] [ 1043.519225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.720940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.745928] LNet: 53238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.759673] LNetError: 53238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.774746] LNet: Removed LNI 10.1.2.3@tcp [ 1043.777861] LNetError: 53238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.805419] LNet: Removed LNI 192.168.204.43@tcp [ 1043.864695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.942633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1043.965652] LNet: Added LNI 192.168.204.43@tcp [8/256/0/182] [ 1043.969432] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1043.972742] LNet: 53246:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1044.203885] LNet: Removed LNI 10.1.2.3@tcp [ 1044.249267] LNet: Removed LNI 192.168.204.43@tcp [ 1045.299862] Key type .llcrypt unregistered [ 1045.315529] Key type ._llcrypt unregistered [ 1060.374905] Key type ._llcrypt registered [ 1060.377572] Key type .llcrypt registered [ 1061.954798] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:16:52 (1790965012) [ 1063.578518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1065.858723] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:16:56 (1790965016) [ 1066.453239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1066.476658] alg: No test for adler32 (adler32-zlib) [ 1067.414834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.459853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1067.484855] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1068.391919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.43 [ 1068.422406] LNet: 54511:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1068.439041] LNet: Removed LNI 192.168.204.43@tcp [ 1075.735694] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:17:06 (1790965026) [ 1076.169721] LNet: 55081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.185594] LNetError: 55081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.207611] LNet: Removed LNI 192.168.204.43@tcp [ 1076.808884] Key type .llcrypt unregistered [ 1076.814718] Key type ._llcrypt unregistered [ 1083.889441] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1084.090678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1085.588658] Key type ._llcrypt registered [ 1085.592177] Key type .llcrypt registered [ 1085.758239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1085.775282] alg: No test for adler32 (adler32-zlib) [ 1086.965608] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1095.114080] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:17:25 (1790965045) [ 1095.468190] LNet: 55985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.476292] LNetError: 55985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.496162] LNet: Removed LNI 10.1.2.3@tcp [ 1096.080560] Key type .llcrypt unregistered [ 1096.086520] Key type ._llcrypt unregistered [ 1105.134355] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.474449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.479448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.485419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1107.246068] Key type ._llcrypt registered [ 1107.248252] Key type .llcrypt registered [ 1107.566969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.607898] alg: No test for adler32 (adler32-zlib) [ 1108.590035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1116.474593] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:17:46 (1790965066) [ 1118.635824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1120.960985] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:17:51 (1790965071) [ 1121.191454] LNet: 57160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.196555] LNetError: 57160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1121.209255] LNet: Removed LNI 10.1.2.3@tcp [ 1121.946691] Key type .llcrypt unregistered [ 1121.956923] Key type ._llcrypt unregistered [ 1130.539428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1130.942372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1130.945842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1130.952846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1133.780615] Key type ._llcrypt registered [ 1133.784365] Key type .llcrypt registered [ 1134.008514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1134.039570] alg: No test for adler32 (adler32-zlib) [ 1135.056965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.143627] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1144.978872] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:18:15 (1790965095) [ 1145.589162] LNet: 58391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.599893] LNetError: 58391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1145.633129] LNet: Removed LNI 10.1.2.3@tcp [ 1146.507676] Key type .llcrypt unregistered [ 1146.509099] Key type ._llcrypt unregistered [ 1153.535519] Key type ._llcrypt registered [ 1153.538888] Key type .llcrypt registered [ 1153.741674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.764402] alg: No test for adler32 (adler32-zlib) [ 1154.778723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1154.814447] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1163.799609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1164.779850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1164.850216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1164.941350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1164.992341] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1165.088259] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1165.494746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.544432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.607614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.674717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.761370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1165.826905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.914660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.965310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.045845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.090609] LNet: 58831:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.126521] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1166.129062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.203702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.263313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.336416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.398830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.453450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.523493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.120736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1167.176136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1167.249406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1167.311649] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1167.671525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.712951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.739961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.770770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.802995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1167.844771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.885467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.916436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.960263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.995422] LNet: 58942:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.019702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.053560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.102774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.131973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.184302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.234090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.303565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1168.908405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1168.993383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1169.053975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1169.386554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.427304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.468323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.501750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.527815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1169.574985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.612392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.645805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.696914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.732981] LNet: 59054:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.756739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.779463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.808685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.863897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.932760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.975090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.030065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.619785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1170.660272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1170.699985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1170.727904] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1170.739251] LNet: Skipped 2 previous similar messages [ 1171.122902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.161648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.220722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.253248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.291978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1171.332624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.385895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.463076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.507763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.574846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.661754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.703075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.744664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.791886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.842202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.873244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.596369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1172.668268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1172.753074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1173.112330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.171714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.221859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.257556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.302524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.345955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.402057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.471921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.552530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.574219] LNet: 59276:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.586178] LNet: 59276:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1173.626794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.686061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.755720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.838737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.897236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.927742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.966303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.604325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1174.648782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1174.708809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1174.745109] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1174.749697] LNet: Skipped 1 previous similar message [ 1175.110062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.166896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.222810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.290962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.357196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1175.384871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.411769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.460638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.498456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.585268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.632242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.694423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.723294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.759978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.805696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.844620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.162632] LNet: 59429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1176.187934] LNetError: 59429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.274782] LNet: Removed LNI 192.168.204.43@tcp [ 1177.769946] Key type .llcrypt unregistered [ 1177.777892] Key type ._llcrypt unregistered [ 1199.891351] Key type ._llcrypt registered [ 1199.895722] Key type .llcrypt registered [ 1201.619981] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:19:11 (1790965151) [ 1202.655612] Key type .llcrypt unregistered [ 1202.657943] Key type ._llcrypt unregistered [ 1210.347916] Key type ._llcrypt registered [ 1210.352719] Key type .llcrypt registered [ 1210.595628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1210.625775] alg: No test for adler32 (adler32-zlib) [ 1211.637132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1211.691553] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1222.085549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1237.978266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.038443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.067977] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1239.099293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1240.020251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1240.079663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 1240.151637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1240.221196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 1240.255725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 1 [ 1240.323726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 1 [ 1240.408744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 1 [ 1240.475636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 1 [ 1240.523907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1240.558310] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1240.562960] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp1 [ 1240.822986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.873423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.914354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.959676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.021419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.070341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.121651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.173192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.208192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.229072] LNet: 60907:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.256996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.327736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.395350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.472059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.550080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.599432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.636868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.137381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1242.167348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 1242.194903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1242.244369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 1242.279543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 1 [ 1242.312549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 1 [ 1242.346691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 1 [ 1242.376387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 1 [ 1242.431519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1242.448803] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1242.454523] LNet: Skipped 2 previous similar messages [ 1242.647404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.671473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.695553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.722111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.765630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1242.819975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.853137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.899658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.937628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.957510] LNet: 61030:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.985441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.025631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.076928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.120485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.178402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.217448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.257469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.774936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1243.817495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 1243.875135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1243.950836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 1244.018849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 1 [ 1244.083241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 1 [ 1244.166719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 1 [ 1244.217350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 1 [ 1244.297527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1244.326464] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1244.331662] LNet: Skipped 2 previous similar messages [ 1244.585829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.626769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.666619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.708971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.767441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1244.812808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.891548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.980802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.045617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.061418] LNet: 61154:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.098219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.175057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.229511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.318526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.399234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.445317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.509488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.154322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1246.182293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 1246.221121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1246.262767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 1246.330216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 1 [ 1246.380198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 1 [ 1246.442207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 1 [ 1246.504652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 1 [ 1246.564777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1246.603401] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1246.612318] LNet: Skipped 4 previous similar messages [ 1246.904111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.965052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.996906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.024952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.064422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1247.112840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.155972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.241233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.310938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.344994] LNet: 61277:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.405307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.452771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.528918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.571486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.600748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.664305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.703901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.499128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1248.531541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 1248.588229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1248.653140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 1248.704583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 1 [ 1248.760443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 1 [ 1248.832591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 1 [ 1248.874398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 1 [ 1248.964142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1249.369819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.444562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.520470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.570771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.632799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.693595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.742649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.779244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.816513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.842964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.875205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.906401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.934326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.969234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.006557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1250.034165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.651840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1250.708791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 1250.754073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1250.865724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 1250.929779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_error -r 1 [ 1250.981993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_error -r 1 [ 1251.019547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_error -r 1 [ 1251.066350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_error -r 1 [ 1251.145299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1251.186164] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1251.190814] LNet: Skipped 5 previous similar messages [ 1251.532298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1251.572438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1251.613433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1251.644773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1251.740410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1251.805454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.826758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.861975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.914254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.954936] LNet: 61524:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.968191] LNet: 61524:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1251.989368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1252.020306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1252.058982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1252.106244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1252.161108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1252.222943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.297922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.480296] LNet: 61600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.491436] LNetError: 61600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.498684] LNet: Removed LNI 192.168.204.43@tcp [ 1260.535218] LNet: Removed LNI 192.168.204.43@tcp1 [ 1261.051664] Key type .llcrypt unregistered [ 1261.063843] Key type ._llcrypt unregistered [ 1280.855763] Key type ._llcrypt registered [ 1280.858248] Key type .llcrypt registered [ 1281.789564] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:20:32 (1790965232) [ 1282.699324] Key type .llcrypt unregistered [ 1282.703735] Key type ._llcrypt unregistered [ 1290.375466] Key type ._llcrypt registered [ 1290.377244] Key type .llcrypt registered [ 1290.813890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1290.852544] alg: No test for adler32 (adler32-zlib) [ 1291.939327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1291.985181] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1302.522724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1303.588760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1303.634434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1303.697432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1303.728621] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1304.024843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.052451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.077788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.101965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.125483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1304.156515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.181900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.208971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.240735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.258139] LNet: 62853:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.272889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.295438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.325435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.380777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.422664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.487430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.527383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.068325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1305.115564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1305.160441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1305.194055] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1305.484803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.526679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.566432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.600231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.639736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.683296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.729177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.774400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.822143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.849370] LNet: 62964:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.878962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.908994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.943221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.973726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.015170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.049825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.089380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.726117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1306.761758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1306.801892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1306.825417] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1307.119598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.170449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.234555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.274891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.333889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1307.384228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.433957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.499884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.573891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.630269] LNet: 63075:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.704964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.768685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.817777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.877652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.918980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.973777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.038226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.419757] LNet: 63117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.430898] LNetError: 63117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.463501] LNet: Removed LNI 192.168.204.43@tcp [ 1309.033642] Key type .llcrypt unregistered [ 1309.036188] Key type ._llcrypt unregistered [ 1329.244405] Key type ._llcrypt registered [ 1329.247633] Key type .llcrypt registered [ 1330.404459] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:21:21 (1790965281) [ 1331.372460] Key type .llcrypt unregistered [ 1331.374104] Key type ._llcrypt unregistered [ 1338.344546] Key type ._llcrypt registered [ 1338.349735] Key type .llcrypt registered [ 1338.545452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1338.566564] alg: No test for adler32 (adler32-zlib) [ 1339.475063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1339.496542] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1348.231616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1360.422258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.494498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1360.552822] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1361.887510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1362.572838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1362.608903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 1362.650633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1362.780917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 1362.827919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 1 [ 1362.890126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 1 [ 1362.987698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 1 [ 1363.045138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 1 [ 1363.116469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1363.156654] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1363.165620] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp1 [ 1363.415859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1363.475156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1363.514105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1363.565799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1363.616953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1363.704863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1363.733962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1363.779979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1363.815891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1363.845629] LNet: 64594:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1363.881078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1363.930948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1364.013613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.074178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1364.139963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.192501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1364.255108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1364.318262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1364.825864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1364.889843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 1364.929529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1364.982044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 1365.049710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_error -r 1 [ 1365.100658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_error -r 1 [ 1365.163788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_error -r 1 [ 1365.224792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_error -r 1 [ 1365.255609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1365.291903] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1365.297904] LNet: Skipped 2 previous similar messages [ 1365.547309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.576284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1365.614406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.642889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1365.669617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1365.703618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1365.725574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1365.749769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1365.775528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1365.793618] LNet: 64720:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.811309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1365.846610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1365.877288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1365.922493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.969411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.003918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1366.053431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.116994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.741227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1366.843565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 1366.914536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1366.989116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 1367.078891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_timeout -r 1 [ 1367.135457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_timeout -r 1 [ 1367.179518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_timeout -r 1 [ 1367.231292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_timeout -r 1 [ 1367.314755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.143@tcp [ 1367.343337] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1367.618879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.672806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1367.714645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.763274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.803237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1367.861744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.913168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.978285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1368.033486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1368.062808] LNet: 64845:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1368.095617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1368.137914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1368.170686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.225238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.283250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.360385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1368.434349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1368.504494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1374.035736] LNet: 64923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.044233] LNetError: 64923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.055759] LNet: Removed LNI 192.168.204.43@tcp [ 1374.072285] LNet: Removed LNI 192.168.204.43@tcp1 [ 1374.687771] Key type .llcrypt unregistered [ 1374.692303] Key type ._llcrypt unregistered [ 1395.329402] Key type ._llcrypt registered [ 1395.331868] Key type .llcrypt registered [ 1397.086794] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:22:27 (1790965347) [ 1398.774866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1400.801224] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:22:31 (1790965351) [ 1401.878397] Key type .llcrypt unregistered [ 1401.880903] Key type ._llcrypt unregistered [ 1408.859937] Key type ._llcrypt registered [ 1408.862595] Key type .llcrypt registered [ 1409.136664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1409.165724] alg: No test for adler32 (adler32-zlib) [ 1410.136612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1410.180608] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1418.384524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1419.340138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1419.423652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1419.524122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1419.572892] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1419.900447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1419.935090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1419.984650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1420.011444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1420.049781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1420.085404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1420.119992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1420.160396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1420.193622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1420.227335] LNet: 66472:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1420.264156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1420.310204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1420.371983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1420.413252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.453064] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1420.458558] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1420.464771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1420.505711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1420.935385] LNet: 66512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.950149] LNetError: 66512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.992891] LNet: Removed LNI 192.168.204.43@tcp [ 1422.494497] Key type .llcrypt unregistered [ 1422.500580] Key type ._llcrypt unregistered [ 1445.046861] Key type ._llcrypt registered [ 1445.049279] Key type .llcrypt registered [ 1445.272403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1445.300482] alg: No test for adler32 (adler32-zlib) [ 1446.319538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1446.347059] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1454.966199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1464.915486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.958068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.984236] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1465.825259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1466.763860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1466.811943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 1466.858602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1466.924910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 1467.001967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e network_timeout -r 1 [ 1467.105271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e network_timeout -r 1 [ 1467.178839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e network_timeout -r 1 [ 1467.235679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e network_timeout -r 1 [ 1467.306081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 1467.329806] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1467.696439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1467.737906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1467.811310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1467.872113] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp1 [ 1467.881553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1467.882583] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp1 [ 1467.922313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1467.950811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1468.000770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1468.059576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1468.153261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1468.191888] LNet: 67400:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.234972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1468.310920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1468.381798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.461053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1468.513806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1468.563870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1474.601995] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1474.897964] LNet: 67474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.904133] LNetError: 67474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.910851] LNet: Removed LNI 192.168.204.43@tcp [ 1474.930617] LNet: Removed LNI 192.168.204.43@tcp1 [ 1475.326559] Key type .llcrypt unregistered [ 1475.329413] Key type ._llcrypt unregistered [ 1494.059762] Key type ._llcrypt registered [ 1494.061838] Key type .llcrypt registered [ 1495.322219] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:24:05 (1790965445) [ 1495.868567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.878947] alg: No test for adler32 (adler32-zlib) [ 1496.766995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.833688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1496.865168] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1497.684267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.716709] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1498.949247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1499.017278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1499.053349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1499.086933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1499.112924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1499.138506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1499.187490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 1499.217557] LNetError: 68573:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1499.872900] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1499.875164] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1501.924547] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1502.804787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1506.017596] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1506.348367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1506.386026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1506.555164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1506.606897] LNet: 68680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.622777] LNetError: 68680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.632527] LNet: Removed LNI 192.168.204.43@tcp [ 1506.653211] LNet: Removed LNI 192.168.204.43@tcp1 [ 1506.706958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.823159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1506.876100] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1507.618462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.639515] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1508.457817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1508.491060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1508.545830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1508.593951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1508.619124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1508.645440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1508.683830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1508.710594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 1508.732642] LNetError: 68953:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 1508.770214] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1511.111446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1514.913584] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1514.921594] LNet: Skipped 1 previous similar message [ 1519.374866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1519.409128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1519.435125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1528.777522] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:24:38 (1790965478) [ 1528.896696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.961329] LNet: 69654:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.976425] LNetError: 69654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.981774] LNetError: 69654:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1528.991981] LNet: Removed LNI 192.168.204.43@tcp [ 1529.016413] LNet: Removed LNI 192.168.204.43@tcp1 [ 1529.069543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.269449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.341819] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1530.345699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1530.379501] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1531.109641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1531.192770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1531.231720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1531.268834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1531.316117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1531.353904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1531.387233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 1531.462413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1532.192518] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.43@tcp [ 1532.193142] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1532.210290] LNet: Skipped 1 previous similar message [ 1546.995271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1547.238694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1547.333509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1547.432973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 500 [ 1547.563487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1554.495532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1554.611520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.656937] LNet: 70153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.666492] LNetError: 70153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.669412] LNetError: 70153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1555.680155] LNet: Removed LNI 192.168.204.43@tcp [ 1555.702855] LNet: Removed LNI 192.168.204.43@tcp1 [ 1555.757724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.858976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.888826] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1556.621298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1556.654434] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1557.723278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1557.880515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1558.030282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1558.106738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1558.182162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1558.247791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1558.380284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1558.459458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.43@tcp --health 0 [ 1559.904424] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1559.918738] LNet: Skipped 3 previous similar messages [ 1561.258256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.997472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1570.111929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1570.229682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1582.133359] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:25:32 (1790965532) [ 1583.135268] LNet: 71190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.148652] LNetError: 71190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.159266] LNetError: 71190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1583.174921] LNet: Removed LNI 192.168.204.43@tcp [ 1583.204718] LNet: Removed LNI 192.168.204.43@tcp1 [ 1584.007153] Key type .llcrypt unregistered [ 1584.010558] Key type ._llcrypt unregistered [ 1592.435351] Key type ._llcrypt registered [ 1592.439649] Key type .llcrypt registered [ 1592.557157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1592.567454] alg: No test for adler32 (adler32-zlib) [ 1593.585349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1593.637118] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1603.513684] Lustre: DEBUG MARKER: Initial discovery [ 1603.726605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1606.312867] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1606.410983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1606.481785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1606.522366] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1606.545953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1606.594363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -l 3 [ 1606.641065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.143@tcp -r 1 -m GET -e local_error [ 1606.671845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.143@tcp -r 1 -m PUT -l 6 [ 1606.944190] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1607.839421] Lustre: DEBUG MARKER: Force oleg443-client.virtnet to discover 192.168.204.143@tcp (in background) [ 1608.031179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.143@tcp [ 1609.208764] Lustre: DEBUG MARKER: Force oleg443-server to discover 192.168.204.43@tcp [ 1611.108469] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1611.249503] Lustre: DEBUG MARKER: Wait for 71954 [ 1612.758514] Lustre: DEBUG MARKER: Finished wait on 71954 [ 1612.807600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1612.857777] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 1612.877816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1613.175867] LNet: 72402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.182297] LNetError: 72402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.201326] LNet: Removed LNI 192.168.204.43@tcp [ 1613.601796] Key type .llcrypt unregistered [ 1613.605129] Key type ._llcrypt unregistered [ 1629.341536] Key type ._llcrypt registered [ 1629.343164] Key type .llcrypt registered [ 1631.357315] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:26:21 (1790965581) [ 1632.042609] Key type .llcrypt unregistered [ 1632.046271] Key type ._llcrypt unregistered [ 1640.297105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1640.452060] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1640.456515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1640.471321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1643.125523] Key type ._llcrypt registered [ 1643.128320] Key type .llcrypt registered [ 1643.407347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1643.439129] alg: No test for adler32 (adler32-zlib) [ 1644.460870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.557632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.618482] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1645.698615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1645.757307] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1654.937608] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:26:45 (1790965605) [ 1655.176570] LNet: 74482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.191072] LNetError: 74482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1655.205863] LNet: Removed LNI 10.1.2.3@tcp [ 1655.221489] LNet: Removed LNI 192.168.204.43@tcp [ 1655.965938] Key type .llcrypt unregistered [ 1655.969735] Key type ._llcrypt unregistered [ 1662.338433] Key type ._llcrypt registered [ 1662.340190] Key type .llcrypt registered [ 1662.461426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1662.473242] alg: No test for adler32 (adler32-zlib) [ 1663.361839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.411692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1663.447299] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1664.129700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1664.162876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1665.545945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.43@tcp [ 1666.592179] LNetError: Unexpected error -22 connecting to 192.168.204.43@tcp at host 192.168.204.43:988 [ 1671.678281] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:27:02 (1790965622) [ 1672.038733] LNet: 75614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.050653] LNetError: 75614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1672.056437] LNet: Removed LNI 10.1.2.3@tcp [ 1672.073032] LNet: Removed LNI 192.168.204.43@tcp [ 1672.672809] Key type .llcrypt unregistered [ 1672.675106] Key type ._llcrypt unregistered [ 1681.532255] Key type ._llcrypt registered [ 1681.534338] Key type .llcrypt registered [ 1681.766236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1681.801235] alg: No test for adler32 (adler32-zlib) [ 1682.809061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.868574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1682.897063] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1683.836902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1683.866678] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 1685.030428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.43@tcp1 --nid 192.168.204.43@tcp2 [ 1690.910459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp2 [ 1690.934323] LNetError: 76449:0:(lib-move.c:2116:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.43@tcp1 to 192.168.204.43@tcp2 - Route table may be misconfigured [ 1690.963428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp2 192.168.204.43@tcp1 [ 1699.307611] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:27:29 (1790965649) [ 1699.427193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.461639] LNet: 77017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.478289] LNetError: 77017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.487397] LNet: Removed LNI 192.168.204.43@tcp1 [ 1700.520850] LNet: Removed LNI 192.168.204.43@tcp2 [ 1700.566754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.623117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1700.641884] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1701.714610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1701.741814] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1702.679389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 1702.707636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.43@tcp [ 1702.734646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp1 192.168.204.43@tcp1 [ 1702.779248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp -e network_timeout [ 1702.852385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -e network_timeout [ 1702.921684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -e network_timeout [ 1702.973285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -e network_timeout [ 1703.025802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1703.063567] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 1703.094582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1703.263827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1703.363378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1703.648328] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 1711.417691] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:27:41 (1790965661) [ 1711.691176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.750166] LNet: 77883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.770970] LNetError: 77883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.782276] LNetError: 77883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1711.796220] LNet: Removed LNI 192.168.204.43@tcp [ 1712.872078] LNet: Removed LNI 192.168.204.43@tcp1 [ 1712.959960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.161247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1713.589960] LNet: 77923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.125640] Key type .llcrypt unregistered [ 1714.129221] Key type ._llcrypt unregistered [ 1727.951512] Key type ._llcrypt registered [ 1727.953852] Key type .llcrypt registered [ 1729.220584] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:27:59 (1790965679) [ 1729.886753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1729.909884] alg: No test for adler32 (adler32-zlib) [ 1730.880403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.411678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1734.209862] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:28:05 (1790965685) [ 1734.737417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.770721] LNet: 79021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.822325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.918855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.951273] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1735.857476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1735.907366] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 1737.028671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1737.084864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp1 [ 1737.128689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp1 [ 1743.239589] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:28:14 (1790965694) [ 1744.996641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg443-server [ 1746.919350] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:28:17 (1790965697) [ 1749.532110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg443-server [ 1752.072578] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:28:21 (1790965701) [ 1754.660326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg443-server [ 1757.329862] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:28:27 (1790965707) [ 1759.848388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg443-server [ 1761.845819] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:28:32 (1790965712) [ 1763.558769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg443-server [ 1765.608353] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:28:36 (1790965716) [ 1768.075572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg443-server [ 1769.951242] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:28:40 (1790965720) [ 1772.323653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg443-server [ 1774.089790] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:28:44 (1790965724) [ 1775.780829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg443-server [ 1777.811775] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:28:48 (1790965728) [ 1780.085533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg443-server [ 1782.697276] Lustre: DEBUG MARKER: == sanity-lnet test 229: Check NI status with lnetctl net disable/enable ========================================================== 14:28:52 (1790965732) [ 1782.924196] LNet: 82897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.933440] LNetError: 82897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.957710] LNet: Removed LNI 192.168.204.43@tcp [ 1783.977664] LNet: Removed LNI 192.168.204.43@tcp1 [ 1784.770630] Key type .llcrypt unregistered [ 1784.774944] Key type ._llcrypt unregistered [ 1791.202198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1791.208499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1793.749725] Key type ._llcrypt registered [ 1793.752429] Key type .llcrypt registered [ 1793.896128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1793.913770] alg: No test for adler32 (adler32-zlib) [ 1794.814190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.893851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.921336] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 1795.563189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1795.585623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1797.603183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if test1pg [ 1797.791396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 1797.821942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.43@tcp [ 1797.847764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 10.1.2.3@tcp [ 1967.072161] INFO: task lnetctl:83671 blocked for more than 120 seconds. [ 1967.076944] Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 1967.080905] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1967.086995] task:lnetctl state:D stack:0 pid:83671 ppid:82756 flags:0x80000000 [ 1967.093808] Call Trace: [ 1967.094979] __schedule+0x351/0xcb0 [ 1967.096478] ? wait_for_completion_timeout+0xa9/0x1d0 [ 1967.099172] schedule+0xc0/0x180 [ 1967.102227] schedule_timeout+0xb4/0x190 [ 1967.105968] ? __next_timer_interrupt+0x160/0x160 [ 1967.109227] ? lnet_send+0xc5/0x270 [lnet] [ 1967.110951] ? __prepare_to_swait+0x5b/0x90 [ 1967.112374] wait_for_completion_timeout+0xeb/0x1d0 [ 1967.118690] lnet_ping+0x1dc/0x840 [lnet] [ 1967.123138] ? lnet_peer_ni_show_start+0x9b0/0x9b0 [lnet] [ 1967.129476] ? lnet_genl_send_scalar_list+0x77/0x230 [lnet] [ 1967.133874] lnet_ping_show_dump+0x270/0xe50 [lnet] [ 1967.139547] ? __alloc_skb+0xc2/0x290 [ 1967.145311] ? __build_skb_around+0xab/0xc0 [ 1967.149638] ? __alloc_skb+0x143/0x290 [ 1967.156141] netlink_dump+0x1d4/0x510 [ 1967.162391] __netlink_dump_start+0x20c/0x2f0 [ 1967.169034] genl_family_rcv_msg_dumpit.isra.12+0x9b/0x170 [ 1967.175027] ? genl_rcv_msg+0xd0/0xd0 [ 1967.179052] ? lnet_discover_nid_metadata+0x1260/0x1260 [lnet] [ 1967.186623] ? genl_rcv+0x60/0x60 [ 1967.188386] genl_family_rcv_msg+0x125/0x1f0 [ 1967.191475] ? lnet_net_show_dump+0x1de0/0x1de0 [lnet] [ 1967.198974] ? lnet_mark_ping_buffer_for_update+0x40/0x40 [lnet] [ 1967.205265] ? lnet_discover_nid_metadata+0x1260/0x1260 [lnet] [ 1967.216818] ? lnet_debug_recovery_show_done+0x310/0x310 [lnet] [ 1967.224813] genl_rcv_msg+0x5b/0xd0 [ 1967.232289] ? genl_family_rcv_msg+0x1f0/0x1f0 [ 1967.236551] netlink_rcv_skb+0x62/0x180 [ 1967.242512] genl_rcv+0x34/0x60 [ 1967.247559] netlink_unicast+0x250/0x3e0 [ 1967.253302] netlink_sendmsg+0x3da/0x600 [ 1967.258829] __sock_sendmsg+0x50/0x90 [ 1967.262951] ____sys_sendmsg+0x1ea/0x330 [ 1967.270646] ? copy_msghdr_from_user+0x74/0xb0 [ 1967.275824] ___sys_sendmsg+0x8c/0xe0 [ 1967.279896] ? do_raw_spin_unlock+0x75/0x190 [ 1967.283363] ? _raw_spin_unlock+0x12/0x30 [ 1967.286961] ? do_fault+0x3f5/0x6a0 [ 1967.289948] ? __handle_mm_fault+0xa51/0xfe0 [ 1967.293316] __sys_sendmsg+0x6b/0xe0 [ 1967.296075] __x64_sys_sendmsg+0x23/0x30 [ 1967.299299] do_syscall_64+0xc1/0x440 [ 1967.307545] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 1967.311214] RIP: 0033:0x7f1a12af4d78 [ 1967.313794] Code: Unable to access opcode bytes at RIP 0x7f1a12af4d4e. [ 1967.317494] RSP: 002b:00007ffd5ca2e7d8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 1967.325127] RAX: ffffffffffffffda RBX: 00000000009aa390 RCX: 00007f1a12af4d78 [ 1967.330391] RDX: 0000000000000000 RSI: 00007ffd5ca2e810 RDI: 0000000000000003 [ 1967.336138] RBP: 00000000009d84b0 R08: 00000000009d8b20 R09: 0000000000000000 [ 1967.350193] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000009aa2a0 [ 1967.354956] R13: 00007ffd5ca2e810 R14: 00007ffd5ca2e910 R15: 00007ffd5ca2e918 [ 1979.381894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if test1pg [ 1979.612577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 10.1.2.3@tcp [ 1979.649673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2 [ 1979.838412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 10.1.2.3@tcp [ 1979.871254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 10.1.2.3@tcp [ 1979.911451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2163.711315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if ens2 [ 2163.838527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2,test1pg [ 2163.993427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2164.023981] LNetError: 83798:0:(lib-move.c:2532:lnet_select_preferred_best_ni()) no path to 192.168.204.43@tcp from net tcp [ 2164.045965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if ens2,test1pg [ 2164.197626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp [ 2164.249800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --if ens2 [ 2164.316579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2 --bogus [ 2164.376468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --if ens2 --net [ 2164.446367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if no_such_if [ 2164.504890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp999 --if ens2 [ 2165.181221] LNet: 83860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.192692] LNetError: 83860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2165.201905] LNet: Removed LNI 10.1.2.3@tcp [ 2166.248879] LNet: Removed LNI 192.168.204.43@tcp [ 2166.748971] Key type .llcrypt unregistered [ 2166.753318] Key type ._llcrypt unregistered [ 2179.769610] Key type ._llcrypt registered [ 2179.773458] Key type .llcrypt registered [ 2181.890483] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:35:31 (1790966131) [ 2182.470337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2182.493528] alg: No test for adler32 (adler32-zlib) [ 2183.552696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.642844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.676121] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2184.774031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2184.850223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2185.079529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.111735] LNet: 84803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.125495] LNetError: 84803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.166382] LNet: Removed LNI 192.168.204.43@tcp [ 2186.185714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.239487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2186.265556] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2187.051779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2187.354865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2187.409072] LNet: 84949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.428639] LNetError: 84949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.454593] LNet: Removed LNI 192.168.204.43@tcp [ 2187.481836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.578665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2187.631953] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2188.663111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2188.825940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2191.204416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.238328] LNet: 85114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.247813] LNetError: 85114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.268648] LNet: Removed LNI 192.168.204.43@tcp [ 2191.305658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2191.351146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2191.373025] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2192.200699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2192.522617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.638822] LNet: Removed LNI 192.168.204.43@tcp [ 2193.710478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.866362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.930702] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2194.859977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2194.974350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2195.327348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.361426] LNet: 85407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.375372] LNet: 85407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2195.384392] LNetError: 85407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.392541] LNetError: 85407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2195.412970] LNet: Removed LNI 192.168.204.43@tcp [ 2195.438909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.572759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2195.612834] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2196.879391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2197.171883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.246387] LNet: Removed LNI 192.168.204.43@tcp [ 2198.333365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.473700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2198.517443] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2199.614930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2199.771653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2200.159953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2200.197496] LNet: 85700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.213865] LNet: 85700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2200.243958] LNetError: 85700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.258401] LNetError: 85700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2201.322439] LNet: Removed LNI 192.168.204.43@tcp [ 2201.394658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.524481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2201.571977] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2202.674790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2202.868467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2203.944897] LNet: Removed LNI 192.168.204.43@tcp [ 2203.968397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.016967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.062473] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2205.262340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2205.476892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2205.910321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.014291] LNet: Removed LNI 192.168.204.43@tcp [ 2207.075795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.193487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2207.225517] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2208.231895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2208.538277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.560611] LNet: 86139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.572612] LNet: 86139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2208.593978] LNetError: 86139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.602806] LNetError: 86139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2209.645365] LNet: Removed LNI 192.168.204.43@tcp [ 2209.680179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.805602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.839248] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2210.628409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2210.784994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2211.096587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.136161] LNet: Removed LNI 192.168.204.43@tcp [ 2212.192583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.289417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2212.323230] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2213.194897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2213.536954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.565990] LNet: Removed LNI 192.168.204.43@tcp [ 2214.608485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.653642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.675921] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2215.480302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2215.554720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.613411] LNet: Removed LNI 192.168.204.43@tcp [ 2215.650540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.694375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.743238] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2216.398911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2216.483871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.528772] LNet: Removed LNI 192.168.204.43@tcp [ 2216.549299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.603699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2216.660854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.703262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.751953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2216.811951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.840607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.874909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2216.936884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2216.966608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.007267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2217.032495] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2217.769698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.810495] LNet: Removed LNI 192.168.204.43@tcp [ 2217.827442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.891077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2225.761930] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:36:16 (1790966176) [ 2226.181773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.264882] LNet: 87455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.276875] LNet: 87455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 2226.336799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.481635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2226.539829] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 2228.342487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2228.382716] LNetError: 87593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.390634] LNetError: 87593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2228.422748] LNet: Removed LNI 192.168.204.43@tcp231 [ 2228.468384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.527640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2228.572856] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 2228.580278] LNet: 87601:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2229.116388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2229.149298] LNet: Removed LNI 192.168.204.43@tcp231 [ 2229.168135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2229.209857] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 2230.590908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2230.630821] LNet: Removed LNI 192.168.204.43@tcp231 [ 2230.650771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.736391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2230.798527] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 2230.805188] LNet: 87754:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2231.200332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2231.230838] LNet: Removed LNI 192.168.204.43@tcp231 [ 2231.242884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2231.272755] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/0] [ 2232.712712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.754235] LNet: Removed LNI 192.168.204.43@tcp231 [ 2232.780248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.854539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2232.877233] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 2232.883671] LNet: 87905:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2233.315914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2233.355599] LNet: Removed LNI 192.168.204.43@tcp231 [ 2233.373145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2233.394453] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/60] [ 2234.642002] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.681580] LNet: Removed LNI 192.168.204.43@tcp231 [ 2234.712858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.803511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2234.907415] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 2234.914545] LNet: 88056:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2235.178919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2235.212161] LNet: Removed LNI 192.168.204.43@tcp231 [ 2235.223820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2235.241659] LNet: Added LNI 192.168.204.43@tcp231 [8/256/0/180] [ 2245.479867] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:36:35 (1790966195) [ 2245.753926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.825117] LNet: Removed LNI 192.168.204.43@tcp231 [ 2245.869682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.988420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2246.016748] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 2246.922811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2247.016286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2247.057465] LNet: Removed LNI 192.168.204.43@tcp232 [ 2247.083919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2247.165663] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 2248.121710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2248.148879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2248.173038] LNet: Removed LNI 192.168.204.43@tcp232 [ 2248.190815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2248.222603] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 2248.941410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2248.977958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2249.017633] LNet: Removed LNI 192.168.204.43@tcp232 [ 2249.045884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2249.085982] LNet: Added LNI 192.168.204.43@tcp232 [8/256/0/180] [ 2249.846265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2249.882111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2249.923732] LNet: Removed LNI 192.168.204.43@tcp232 [ 2249.935792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2255.989942] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:36:46 (1790966206) [ 2257.296884] Key type .llcrypt unregistered [ 2257.301548] Key type ._llcrypt unregistered [ 2263.168354] Key type ._llcrypt registered [ 2263.170957] Key type .llcrypt registered [ 2263.338862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2263.347453] alg: No test for adler32 (adler32-zlib) [ 2264.203501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2264.224298] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2272.555603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2280.962609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.043090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2281.071197] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 2281.856742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2282.499900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 2282.530192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 2282.566788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 2282.615893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 2282.671845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_interrupt -r 2 [ 2282.706850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_interrupt -r 2 [ 2282.764535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_interrupt -r 2 [ 2282.820178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_interrupt -r 2 [ 2282.865674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2282.884994] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 2282.899212] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 2283.204990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2283.256218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2283.337294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2283.375855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2283.416732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2283.450833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2283.483785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2283.525650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2283.560025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2283.598683] LNet: 90560:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.628347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2283.665905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2283.701384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2283.734893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2283.779381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2283.809234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2284.373234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 2284.403509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 2284.444542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 2284.487155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 2284.537443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_dropped -r 2 [ 2284.589655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_dropped -r 2 [ 2284.651268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_dropped -r 2 [ 2284.708577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_dropped -r 2 [ 2284.743866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2284.770444] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 2285.034851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2285.069457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2285.097473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2285.126859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2285.155900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2285.213389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2285.253879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2285.278069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2285.325456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2285.350262] LNet: 90682:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.379879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2285.428349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2285.472907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2285.520325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2285.559853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.597086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.100532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 2286.132209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 2286.174323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 2286.246994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 2286.311872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_aborted -r 2 [ 2286.384894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_aborted -r 2 [ 2286.464410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_aborted -r 2 [ 2286.508149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_aborted -r 2 [ 2286.579065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2286.623541] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 2286.956981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2287.003622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2287.064039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2287.121111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2287.163499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2287.214422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2287.263432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2287.316048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2287.353294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2287.375240] LNet: 90804:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2287.390618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2287.413829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2287.447677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2287.474723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2287.502162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.538279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2288.059425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 2288.125145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 2288.172631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 2288.212500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 2288.254501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_no_route -r 2 [ 2288.300661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_no_route -r 2 [ 2288.339377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_no_route -r 2 [ 2288.377267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_no_route -r 2 [ 2288.409778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2288.773697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.827610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.876405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.913958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.966534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2289.006950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.053264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.099809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.145348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.191810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2289.242258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2289.279793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2289.340718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2289.372553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2289.411886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.087648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 2290.149471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 2290.195879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 2290.253471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 2290.314187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e local_timeout -r 2 [ 2290.349403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e local_timeout -r 2 [ 2290.398557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e local_timeout -r 2 [ 2290.424701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e local_timeout -r 2 [ 2290.452731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2290.472920] LNet: There was an unexpected network error while writing to 192.168.204.143: rc = -22 [ 2290.482208] LNet: Skipped 1 previous similar message [ 2290.737542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2290.770832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2290.803350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2290.859317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2290.894426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2290.922951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2290.956283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.035946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.105379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.161579] LNet: 91049:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.172391] LNet: 91049:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2291.207413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.271241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.306926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.331471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.353846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.380941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2291.960661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 2292.014194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 2292.061467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 2292.113768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 2292.148450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp -m GET -e remote_dropped -r 2 [ 2292.180937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.143@tcp1 -m GET -e remote_dropped -r 2 [ 2292.227484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp -m GET -e remote_dropped -r 2 [ 2292.262959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.43@tcp1 -d 192.168.204.43@tcp1 -m GET -e remote_dropped -r 2 [ 2292.293232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.43@tcp 192.168.204.143@tcp [ 2292.318387] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 2292.556144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2292.603030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2292.645705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2292.694568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2292.748691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2292.806759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2292.842969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2292.893548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2292.942237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2292.981480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2293.019586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.047385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2293.089971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2293.136740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.165518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2298.497623] LNet: 91245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.507583] LNetError: 91245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.516810] LNet: Removed LNI 192.168.204.43@tcp [ 2298.547359] LNet: Removed LNI 192.168.204.43@tcp1 [ 2299.171425] Key type .llcrypt unregistered [ 2299.177347] Key type ._llcrypt unregistered [ 2314.635220] Key type ._llcrypt registered [ 2314.639610] Key type .llcrypt registered [ 2315.718909] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:37:46 (1790966266) [ 2316.307663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.331223] alg: No test for adler32 (adler32-zlib) [ 2317.282297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.324733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.358759] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2318.385194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2318.437512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.43@tcp [ 2318.472448] LNetError: 92215:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 2318.604588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.43@tcp [ 2318.734521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2327.825786] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:37:58 (1790966278) [ 2328.174911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.190193] LNet: 92805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.197293] LNetError: 92805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.217703] LNet: Removed LNI 192.168.204.43@tcp [ 2328.236468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.305653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.338639] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2329.045789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2329.061480] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 2330.465867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.43@tcp [ 2330.491404] LNetError: 93071:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.43@tcp to recovery [ 2331.360630] LNet: 1 local NIs in recovery (showing 1): 192.168.204.43@tcp [ 2331.375163] LNet: There was an unexpected network error while writing to 192.168.204.43: rc = -22 [ 2331.866728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.43@tcp [ 2342.081688] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:38:12 (1790966292) [ 2344.257382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg443-server [ 2346.449431] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:38:17 (1790966297) [ 2348.773883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg443-server [ 2350.545201] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:38:21 (1790966301) [ 2352.545481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg443-server [ 2354.089305] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:38:25 (1790966305) [ 2354.371227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.396148] LNet: 94701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.404884] LNetError: 94701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2354.411909] LNet: Removed LNI 192.168.204.43@tcp [ 2354.431542] LNet: Removed LNI 192.168.204.43@tcp1 [ 2354.458970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.520565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.540109] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2355.141052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2355.237286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2355.291658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2355.404095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2355.465223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2355.544632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2360.951795] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:38:31 (1790966311) [ 2361.071641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.104766] LNet: 95431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.115828] LNetError: 95431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.124263] LNetError: 95431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2361.141369] LNet: Removed LNI 192.168.204.43@tcp [ 2361.162600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.203559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2361.219942] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2367.933311] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:38:38 (1790966318) [ 2368.463694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.493382] LNet: 96135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.508153] LNetError: 96135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.528474] LNet: Removed LNI 192.168.204.43@tcp [ 2368.549948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.642299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.686115] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2375.538234] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:38:46 (1790966326) [ 2376.892240] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2378.965820] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:38:49 (1790966329) [ 2379.792989] LNet: 97170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.804868] LNetError: 97170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.830431] LNet: Removed LNI 192.168.204.43@tcp [ 2380.479586] Key type .llcrypt unregistered [ 2380.482267] Key type ._llcrypt unregistered [ 2387.098226] Key type ._llcrypt registered [ 2387.103939] Key type .llcrypt registered [ 2387.301559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2387.341993] alg: No test for adler32 (adler32-zlib) [ 2388.306632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2388.334912] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2398.211049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2403.531984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.143@tcp [ 2404.003322] LNet: 97581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2404.010522] LNetError: 97581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2404.032315] LNet: Removed LNI 192.168.204.43@tcp [ 2404.443458] Key type .llcrypt unregistered [ 2404.450562] Key type ._llcrypt unregistered [ 2416.376378] Key type ._llcrypt registered [ 2416.379787] Key type .llcrypt registered [ 2417.940631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2419.741171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2421.091562] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:39:32 (1790966372) [ 2422.768556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2424.441962] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:39:35 (1790966375) [ 2426.284866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg443-server [ 2428.397803] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:39:38 (1790966378) [ 2430.080641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg443-server [ 2431.928337] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:39:42 (1790966382) [ 2433.544266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.590405] alg: No test for adler32 (adler32-zlib) [ 2434.710120] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2435.035033] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2436.822842] Key type lgssc registered [ 2438.314089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.925489] Key type lgssc unregistered [ 2459.186574] LNet: 100751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.207047] LNetError: 100751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.238658] LNet: Removed LNI 192.168.204.43@tcp [ 2459.944650] Key type .llcrypt unregistered [ 2459.947401] Key type ._llcrypt unregistered [ 2478.746249] Key type ._llcrypt registered [ 2478.750289] Key type .llcrypt registered [ 2480.121313] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:40:30 (1790966430) [ 2480.659421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.674204] alg: No test for adler32 (adler32-zlib) [ 2481.607739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.685983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2481.747826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2481.814884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2481.886050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2491.059795] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:40:40 (1790966440) [ 2491.402972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.472292] LNet: 102166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.521522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2491.644192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2491.752836] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2493.831562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2493.886903] LNet: 102311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.898235] LNetError: 102311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.925447] LNet: Removed LNI 192.168.204.43@tcp [ 2493.973499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.040498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2494.077823] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2494.598851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.612381] LNetError: 102339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2494.628815] LNet: Removed LNI 192.168.204.43@tcp [ 2494.650709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.719513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2494.743289] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2503.644226] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:40:54 (1790966454) [ 2503.858930] LNet: 102929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2503.863798] LNet: 102929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2503.871767] LNetError: 102929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2503.890919] LNet: Removed LNI 192.168.204.43@tcp [ 2504.484214] Key type .llcrypt unregistered [ 2504.486718] Key type ._llcrypt unregistered [ 2511.464216] Key type ._llcrypt registered [ 2511.467174] Key type .llcrypt registered [ 2511.819512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2511.864244] alg: No test for adler32 (adler32-zlib) [ 2512.990581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.224288] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2513.226056] LNet: *** cfs_fail_loc=e003, val=100*** [ 2514.202158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2514.233182] LNet: *** cfs_fail_loc=e003, val=100*** [ 2514.240432] LNet: Skipped 2 previous similar messages [ 2514.280966] Ping reply received with mapping: [ 2514.285862] found metadata reply NI[192.168.204.43@tcp] [ 2514.289897] 2b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2514.292787] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2514.296117] 5a 5a 5a 5a 5a 5a 5a 5a [ 2514.493758] LNet: 103320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.524515] LNetError: 103320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2514.543973] LNet: Removed LNI 192.168.204.43@tcp [ 2515.234695] Key type .llcrypt unregistered [ 2515.237382] Key type ._llcrypt unregistered [ 2530.070841] Key type ._llcrypt registered [ 2530.073725] Key type .llcrypt registered [ 2531.663842] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:41:21 (1790966481) [ 2532.583450] Key type .llcrypt unregistered [ 2532.587890] Key type ._llcrypt unregistered [ 2542.579807] Key type ._llcrypt registered [ 2542.581619] Key type .llcrypt registered [ 2543.062525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2543.112165] alg: No test for adler32 (adler32-zlib) [ 2544.595535] LNetError: 104350:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2545.157419] Key type .llcrypt unregistered [ 2545.159792] Key type ._llcrypt unregistered [ 2552.876556] Key type ._llcrypt registered [ 2552.878442] Key type .llcrypt registered [ 2554.449461] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:41:44 (1790966504) [ 2555.730192] Key type .llcrypt unregistered [ 2555.732312] Key type ._llcrypt unregistered [ 2564.816380] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2565.016875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2565.023704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2565.030490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2567.693456] Key type ._llcrypt registered [ 2567.698182] Key type .llcrypt registered [ 2567.962707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2567.981379] alg: No test for adler32 (adler32-zlib) [ 2568.962287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2569.048966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2580.244386] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:42:10 (1790966530) [ 2580.562741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2580.619401] LNet: 106140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.637521] LNetError: 106140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2580.663502] LNet: Removed LNI 10.1.2.3@tcp [ 2580.707208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2581.617512] Key type .llcrypt unregistered [ 2581.621564] Key type ._llcrypt unregistered [ 2588.924698] Key type ._llcrypt registered [ 2588.927806] Key type .llcrypt registered [ 2589.180964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2589.206569] alg: No test for adler32 (adler32-zlib) [ 2590.209664] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2590.290697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2590.465638] LNet: 106427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2590.474603] LNetError: 106427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2590.513880] LNet: Removed LNI 192.168.204.43@tcp [ 2591.163718] Key type .llcrypt unregistered [ 2591.180875] Key type ._llcrypt unregistered [ 2603.697810] Key type ._llcrypt registered [ 2603.701129] Key type .llcrypt registered [ 2605.070334] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:42:35 (1790966555) [ 2606.065496] Key type .llcrypt unregistered [ 2606.067323] Key type ._llcrypt unregistered [ 2613.709150] Key type ._llcrypt registered [ 2613.711708] Key type .llcrypt registered [ 2613.917498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2613.930317] alg: No test for adler32 (adler32-zlib) [ 2621.750724] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:42:52 (1790966572) [ 2621.857712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2621.909752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2621.970632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2622.022250] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2622.753753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2622.813224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2622.847669] LNet: Removed LNI 192.168.204.43@tcp [ 2623.247279] LNet: 108172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2623.656474] Key type .llcrypt unregistered [ 2623.662207] Key type ._llcrypt unregistered [ 2635.821639] Key type ._llcrypt registered [ 2635.825088] Key type .llcrypt registered [ 2637.143651] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:43:07 (1790966587) [ 2638.043594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.077352] alg: No test for adler32 (adler32-zlib) [ 2639.013877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2639.088309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2639.109361] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2639.835711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2639.874895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.43@tcp [ 2639.893387] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2639.897799] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2639.901722] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2639, 0, 0 [ 2639.907854] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2639.910405] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2639.917593] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2639.922240] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2639, 0, 0 [ 2639.930474] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2639.935178] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2639.938966] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2639.943319] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2639, 0, 2689 [ 2639.948360] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2639.951637] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2639.954910] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2639.957317] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2639, 0, 2689 [ 2639.964181] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2646.499542] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:43:17 (1790966597) [ 2646.870875] LNet: 109712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.883872] LNetError: 109712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.922116] LNet: Removed LNI 192.168.204.43@tcp [ 2648.466331] Key type .llcrypt unregistered [ 2648.472241] Key type ._llcrypt unregistered [ 2654.987439] Key type ._llcrypt registered [ 2654.991097] Key type .llcrypt registered [ 2655.232182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2655.259681] alg: No test for adler32 (adler32-zlib) [ 2656.165296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2656.188933] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2664.287547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2675.096937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2675.195917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2675.239073] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 2676.481683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2676.673808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2676.779069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2676.781595] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2676.807314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2679.213399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2679.241273] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2694.528862] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:44:04 (1790966644) [ 2694.883177] LNet: 111223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2694.897755] LNetError: 111223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.968604] LNet: Removed LNI 192.168.204.43@tcp [ 2695.975212] LNetError: 111223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2695.996438] LNet: Removed LNI 192.168.204.43@tcp1 [ 2696.019580] LNet: Removed LNI 10.1.2.3@tcp99 [ 2696.961318] Key type .llcrypt unregistered [ 2696.963180] Key type ._llcrypt unregistered [ 2705.051697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2705.062940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2707.766829] Key type ._llcrypt registered [ 2707.774804] Key type .llcrypt registered [ 2708.029504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2708.046635] alg: No test for adler32 (adler32-zlib) [ 2709.036082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2709.171127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2709.266111] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2710.354170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2710.372489] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2712.562341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2712.620698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2712.691662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2712.719655] LNetError: 111967:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2712.746335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2712.779407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2713.092688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2713.200381] LNet: 111987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2713.210367] LNetError: 111987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2713.218646] LNet: Removed LNI 10.1.2.3@tcp [ 2714.284884] LNet: Removed LNI 192.168.204.43@tcp [ 2715.142395] Key type .llcrypt unregistered [ 2715.144066] Key type ._llcrypt unregistered [ 2730.816856] Key type ._llcrypt registered [ 2730.821045] Key type .llcrypt registered [ 2732.529298] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:44:42 (1790966682) [ 2733.214274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2733.261168] alg: No test for adler32 (adler32-zlib) [ 2734.322585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2734.400272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2734.430592] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2735.380716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2735.479862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg443-client@tcp [ 2744.900728] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:44:55 (1790966695) [ 2744.993250] LNet: 113511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2745.004316] LNetError: 113511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2745.031072] LNet: Removed LNI 192.168.204.43@tcp [ 2745.816395] Key type .llcrypt unregistered [ 2745.819776] Key type ._llcrypt unregistered [ 2753.255429] Key type ._llcrypt registered [ 2753.258457] Key type .llcrypt registered [ 2753.422299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2753.463500] alg: No test for adler32 (adler32-zlib) [ 2754.325236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2754.356884] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2754.385689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2760.768700] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:45:11 (1790966711) [ 2762.442979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2763.908709] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:45:14 (1790966714) [ 2765.218988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2766.962536] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:45:17 (1790966717) [ 2767.175547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2767.204830] LNet: 114942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2767.210644] LNetError: 114942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2767.228852] LNet: Removed LNI 192.168.204.43@tcp [ 2767.248600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2767.295724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2767.340464] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2768.303457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2768.374913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2775.311440] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:45:26 (1790966726) [ 2775.676951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2775.733444] LNet: 115652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.748280] LNetError: 115652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.769335] LNet: Removed LNI 192.168.204.43@tcp [ 2775.797586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2775.931853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2775.975348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2776.638625] Key type .llcrypt unregistered [ 2776.640109] Key type ._llcrypt unregistered [ 2790.381823] Key type ._llcrypt registered [ 2790.383758] Key type .llcrypt registered [ 2792.255994] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:45:42 (1790966742) [ 2792.838291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2792.862482] alg: No test for adler32 (adler32-zlib) [ 2793.851198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2793.914140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2793.948324] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2795.477978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2795.630215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2805.392836] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:45:55 (1790966755) [ 2805.468579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2805.493342] LNet: 117202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2805.499264] LNetError: 117202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.512544] LNet: Removed LNI 192.168.204.43@tcp [ 2805.539201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2805.588761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2805.616409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2813.317902] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:46:04 (1790966764) [ 2813.508846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2813.553242] LNet: 117787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2813.586891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2813.644826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2814.352513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2814.408580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2814.472821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2814.902668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2814.932906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2815.008353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2815.038991] LNet: 117821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.079413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2815.188685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2815.627732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2815.658766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2815.714468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2816.206434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2816.245538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2823.041148] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:46:13 (1790966773) [ 2823.156053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2823.189712] LNet: 118415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2823.230156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2823.297382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2823.324693] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2823.940843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2823.956703] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2824.896831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2824.967082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2825.289534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2825.326967] LNetError: 118701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.336171] LNet: Removed LNI 192.168.204.43@tcp [ 2826.341611] LNetError: 118701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.354500] LNet: Removed LNI 192.168.204.43@tcp2 [ 2826.398376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2826.485076] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2826.492812] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2826.497171] LNet: 118707:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2827.169685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2827.507294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2827.547630] LNet: 118744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.561919] LNet: 118744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2828.576313] LNet: Removed LNI 192.168.204.43@tcp [ 2828.579375] LNetError: 118744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2828.588612] LNetError: 118744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2828.602072] LNet: Removed LNI 192.168.204.43@tcp2 [ 2828.661556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2828.716563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2828.740844] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2829.471294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2829.489170] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2830.893797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2830.942885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2831.017903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2831.333606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2832.418356] LNet: Removed LNI 192.168.204.43@tcp [ 2833.451535] LNet: Removed LNI 192.168.204.43@tcp2 [ 2833.471961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2833.566365] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2833.580277] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2833.608094] LNet: 119035:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2834.094540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2834.450360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2834.471534] LNetError: 119072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2834.477807] LNetError: 119072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2834.485451] LNet: Removed LNI 192.168.204.43@tcp [ 2835.498078] LNet: Removed LNI 192.168.204.43@tcp2 [ 2835.515887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2835.551960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2835.581526] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2836.226785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2836.261596] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2837.135399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2837.228728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2837.493246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2837.509932] LNet: 119352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2837.514958] LNet: 119352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2838.560372] LNet: Removed LNI 192.168.204.43@tcp [ 2838.572857] LNet: Removed LNI 192.168.204.43@tcp2 [ 2838.627694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2838.747967] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2838.751819] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2838.754871] LNet: 119358:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2839.091362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2839.400877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2840.480343] LNet: Removed LNI 192.168.204.43@tcp [ 2840.494844] LNet: Removed LNI 192.168.204.43@tcp2 [ 2840.568652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2840.677561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2840.722534] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2841.672935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2841.746296] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2843.204658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2843.326912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2843.474527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2843.906940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2843.965646] LNetError: 119680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2843.971300] LNetError: 119680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2843.977569] LNet: Removed LNI 192.168.204.43@tcp [ 2845.030401] LNet: Removed LNI 192.168.204.43@tcp2 [ 2845.068630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2845.224766] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2845.232290] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2845.240295] LNet: 119686:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2845.819893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2846.300966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2846.342930] LNet: Removed LNI 192.168.204.43@tcp [ 2847.398338] LNet: Removed LNI 192.168.204.43@tcp2 [ 2847.454994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2847.649408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2847.695808] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2848.620836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2848.642795] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2849.657976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2849.702673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2849.776390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2850.197741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2851.234950] LNet: Removed LNI 192.168.204.43@tcp [ 2851.256223] LNet: Removed LNI 192.168.204.43@tcp2 [ 2851.312168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2851.378144] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2851.388813] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2851.393455] LNet: 120026:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2851.877338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2852.164913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2853.219242] LNet: Removed LNI 192.168.204.43@tcp [ 2853.237182] LNet: Removed LNI 192.168.204.43@tcp2 [ 2853.301655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2853.423522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2853.456343] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2854.592734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2854.629179] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2856.143496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2856.266167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2856.292780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2856.415456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2856.747463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2856.761118] LNet: 120379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2856.764813] LNet: 120379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2857.824179] LNet: Removed LNI 192.168.204.43@tcp [ 2858.859270] LNet: Removed LNI 192.168.204.43@tcp2 [ 2859.001822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2859.188478] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2859.209341] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 2860.180520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2869.894748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2871.505840] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:47:02 (1790966822) [ 2872.031931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2872.055750] LNetError: 121146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2872.066089] LNetError: 121146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2872.070081] LNet: Removed LNI 192.168.204.43@tcp [ 2873.130469] LNet: Removed LNI 192.168.204.43@tcp2 [ 2873.179356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2873.468262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2873.522424] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2874.606209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2874.654612] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2875.850920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2875.918146] LNet: Removed LNI 10.1.2.3@tcp [ 2875.957586] LNet: Removed LNI 192.168.204.43@tcp [ 2876.024442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2876.111721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2876.132145] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2876.137268] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2876.494309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2876.538708] LNet: Removed LNI 192.168.204.43@tcp [ 2876.570626] LNet: Removed LNI 10.1.2.3@tcp [ 2884.159750] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 14:47:14 (1790966834) [ 2884.302245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2884.365377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2884.470666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2884.534767] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2885.608290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2885.669227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2885.776567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2885.910136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2885.985577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.053590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.127916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.219057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.296067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.406472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.530822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2886.594714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2887.025156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2897.776278] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 14:47:27 (1790966847) [ 2898.115436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2898.170770] LNet: 122762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2898.190788] LNet: 122762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2899.263384] LNet: Removed LNI 192.168.204.43@tcp [ 2899.322521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2899.521302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2899.596803] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2901.219397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2901.311960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2901.359990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.416808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.461591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.507702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.549692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.595717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.637655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.670176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.709874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.741837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.845935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2901.903934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.944149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2901.977776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.028226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.081183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.142798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.211729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.257861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.330847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2902.387744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.43@tcp [ 2913.803933] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:47:43 (1790966863) [ 2913.973426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2914.040355] LNetError: 123513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.056305] LNetError: 123513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2914.089168] LNet: Removed LNI 192.168.204.43@tcp [ 2914.147918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2914.372667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2914.735526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2914.742608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2914.745433] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2917.255673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2917.321244] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2918.355980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2918.396515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2921.401251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 2921.464272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2921.520946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.43@tcp -m PUT -e local_timeout -r 1 [ 2921.585559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2921.664782] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2921.667389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2921.942593] LNetError: 123581:0:(lib-move.c:4823:lnet_parse()) 192.168.204.43@tcp, src 192.168.204.43@tcp: Dropping PUT (error -108 looking up sender) [ 2922.978526] LNet: Removed LNI 10.1.2.3@tcp [ 2924.041809] LNet: Removed LNI 192.168.204.43@tcp [ 2924.613507] Key type .llcrypt unregistered [ 2924.619233] Key type ._llcrypt unregistered [ 2937.396736] Key type ._llcrypt registered [ 2937.400396] Key type .llcrypt registered [ 2938.649186] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:48:09 (1790966889) [ 2939.285907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2939.312748] alg: No test for adler32 (adler32-zlib) [ 2940.332164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2940.747177] LNet: 124894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2941.357457] Key type .llcrypt unregistered [ 2941.361432] Key type ._llcrypt unregistered [ 2948.096642] Key type ._llcrypt registered [ 2948.106094] Key type .llcrypt registered [ 2948.369796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2948.410531] alg: No test for adler32 (adler32-zlib) [ 2949.387196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2949.416798] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 2957.253211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 2957.504720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2958.255142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 2958.593632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2958.628610] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 2958.647863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2958.684248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2958.741905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2958.797103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2958.832100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2958.918946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2959.001077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2959.084434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2959.104515] LNet: 125345:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2959.147455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2959.212568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2959.358871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2959.406973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2959.470876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2959.546510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2960.526473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2960.983052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2961.025686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2961.111997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2961.170926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2961.224794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2961.291627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2961.352490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2961.413321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2961.513280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2961.563077] LNet: 125465:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2961.606491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2961.670587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2961.728984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2961.777562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2961.850796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2961.914972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2962.885929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 2963.215852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2963.252562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2963.282784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2963.314324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2963.344330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2963.397715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2963.447788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2963.554818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2963.623719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2963.649887] LNet: 125584:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2963.695748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2963.769585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2963.831468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2963.876832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2963.938576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2964.010315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2965.094811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2965.544637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2965.593010] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2965.624903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2965.679615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2965.754579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2965.840334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2965.960906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2966.064807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2966.151777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2966.191103] LNet: 125704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2966.254655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2966.335804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2966.401985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2966.486645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2966.565181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2966.638341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2967.074598] LNet: 125750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.091588] LNetError: 125750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.138649] LNet: Removed LNI 192.168.204.43@tcp [ 2967.763745] Key type .llcrypt unregistered [ 2967.767496] Key type ._llcrypt unregistered [ 2992.847477] Key type ._llcrypt registered [ 2992.851755] Key type .llcrypt registered [ 2994.328683] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:49:04 (1790966944) [ 2994.664701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2994.673312] alg: No test for adler32 (adler32-zlib) [ 2995.571030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2995.926316] LNet: 126637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.449579] Key type .llcrypt unregistered [ 2996.455530] Key type ._llcrypt unregistered [ 3003.402439] Key type ._llcrypt registered [ 3003.405120] Key type .llcrypt registered [ 3003.587704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3003.599797] alg: No test for adler32 (adler32-zlib) [ 3004.493218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3004.514813] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 3012.541967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 3023.787864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3023.876866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3023.920119] LNet: Added LNI 192.168.204.43@tcp1 [8/256/0/180] [ 3025.246399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.143@tcp [ 3025.499764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3025.939882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.143@tcp [ 3026.016496] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.143@tcp [ 3026.181803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3026.210709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3026.241848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3026.289900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3026.336087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3026.385719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3026.426637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3026.492896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3026.528847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3026.562596] LNet: 127296:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3026.584879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3026.686643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3026.761363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3026.802785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3026.866457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3026.931786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3028.177270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3028.754225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3028.865931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3028.969648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3029.005539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3029.108903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3029.211477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3029.309385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3029.361176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3029.434970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3029.481421] LNet: 127416:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3029.543539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3029.611988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3029.702978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3029.787332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3029.884246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3029.976107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3030.730623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.143@tcp [ 3031.047426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3031.095807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3031.149302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3031.233577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3031.300472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3031.358530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3031.451487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3031.488237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3031.527709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3031.568289] LNet: 127535:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3031.596677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3031.632956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3031.708713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3031.742559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3031.781422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3031.829512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3032.935607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3033.250924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3033.294968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3033.346601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3033.400416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3033.449288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3033.543749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3033.628340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3033.725931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3033.826120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3033.872839] LNet: 127655:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3033.922164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3034.011502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3034.074330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3034.175895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3034.247720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3034.301501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3040.976787] LNet: 127736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3040.987711] LNetError: 127736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3040.995153] LNet: Removed LNI 192.168.204.43@tcp [ 3041.010724] LNet: Removed LNI 192.168.204.43@tcp1 [ 3041.405673] Key type .llcrypt unregistered [ 3041.412380] Key type ._llcrypt unregistered [ 3060.649105] Key type ._llcrypt registered [ 3060.653067] Key type .llcrypt registered [ 3062.783784] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 14:50:12 (1790967012) [ 3063.487990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3063.530129] alg: No test for adler32 (adler32-zlib) [ 3064.584624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3064.694985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3064.738301] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 3065.681215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3065.704071] LNet: Added LNI 192.168.204.43@tcp2 [8/256/0/180] [ 3067.080641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.43@tcp [ 3067.239516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3068.321112] LNet: Removed LNI 192.168.204.43@tcp [ 3077.104651] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:50:27 (1790967027) [ 3078.965794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3081.364127] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:50:31 (1790967031) [ 3090.021838] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:50:40 (1790967040) [ 3098.609630] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:50:48 (1790967048) [ 3107.959569] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:50:58 (1790967058) [ 3120.199261] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:51:10 (1790967070) [ 3128.754023] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:51:19 (1790967079) [ 3137.294723] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:51:28 (1790967088) [ 3146.085521] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:51:36 (1790967096) [ 3153.721419] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:51:44 (1790967104) [ 3163.490199] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:51:54 (1790967114) [ 3175.564894] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:52:05 (1790967125) [ 3185.383046] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:52:15 (1790967135) [ 3193.540487] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:52:23 (1790967143) [ 3201.495839] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:52:32 (1790967152) [ 3210.170514] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:52:40 (1790967160) [ 3218.157049] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:52:48 (1790967168) [ 3226.520079] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:52:56 (1790967176) [ 3233.953423] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:53:04 (1790967184) [ 3234.211426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3234.240905] LNet: 139434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3234.248244] LNetError: 139434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3234.268231] LNet: Removed LNI 192.168.204.43@tcp2 [ 3234.289161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3234.442491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3234.477754] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 3235.702899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3235.745524] LNet: 139576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3235.753981] LNetError: 139576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3235.789285] LNet: Removed LNI 192.168.204.43@tcp [ 3235.825624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3235.934771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3235.975744] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 3244.893115] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:53:15 (1790967195) [ 3254.381778] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:53:25 (1790967205) [ 3264.820784] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:53:34 (1790967214) [ 3277.599980] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:53:47 (1790967227) [ 3285.724480] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:53:56 (1790967236) [ 3293.940616] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:54:04 (1790967244) [ 3304.309933] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:54:14 (1790967254) [ 3314.592586] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:54:24 (1790967264) [ 3325.766895] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:54:36 (1790967276) [ 3336.657379] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:54:46 (1790967286) [ 3345.817542] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:54:56 (1790967296) [ 3352.677917] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:55:03 (1790967303) [ 3360.708614] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3100 sec ========== 14:55:11 (1790967311) [ 3360.927476] LNet: 147045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3360.934768] LNetError: 147045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3360.957873] LNet: Removed LNI 192.168.204.43@tcp [ 3361.622188] Key type .llcrypt unregistered [ 3361.628656] Key type ._llcrypt unregistered [ 3374.737637] Key type ._llcrypt registered [ 3374.755806] Key type .llcrypt registered [ 3375.280512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3375.304746] alg: No test for adler32 (adler32-zlib) [ 3376.565939] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3376.975112] LNet: Added LNI 192.168.204.43@tcp [8/256/0/180] [ 3378.656261] Key type lgssc registered [ 3379.752069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3460.769667] Lustre: Mounted lustre-client - version 2.17.59_2_g3d9f037 [ 3467.168766] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3478.441651] Lustre: Unmounted lustre-client [ 3517.979306] Key type lgssc unregistered [ 3518.424536] LNet: 150008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.455887] LNetError: 150008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.486510] LNet: Removed LNI 192.168.204.43@tcp [ 3519.480177] Key type .llcrypt unregistered [ 3519.486446] Key type ._llcrypt unregistered