[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494472559 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003342] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007991] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.010159] LSM: Security Framework initializing [ 0.011090] Yama: becoming mindful. [ 0.012072] SELinux: Initializing. [ 0.013081] *** VALIDATE selinux *** [ 0.021217] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026046] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027185] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029135] *** VALIDATE tmpfs *** [ 0.031475] *** VALIDATE proc *** [ 0.033078] *** VALIDATE cgroup *** [ 0.034015] *** VALIDATE cgroup2 *** [ 0.036173] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039039] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043533] debug: unmapping init [mem 0xffffffff92659000-0xffffffff92660fff] [ 0.045186] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046704] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047032] ... version: 2 [ 0.048018] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051018] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053015] ... event mask: 000000070000000f [ 0.054331] rcu: Hierarchical SRCU implementation. [ 0.056677] smp: Bringing up secondary CPUs ... [ 0.057713] x86: Booting SMP configuration: [ 0.058040] .... node #0, CPUs: #1 #2 #3 [ 0.062024] smp: Brought up 1 node, 4 CPUs [ 0.064022] smpboot: Max logical packages: 1 [ 0.065024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.120130] node 0 deferred pages initialised in 51ms [ 0.124241] devtmpfs: initialized [ 0.125287] x86/mm: Memory block size: 128MB [ 0.127842] gcov: version magic: 0x41383552 [ 0.129387] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.130124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.131323] pinctrl core: initialized pinctrl subsystem [ 0.132277] [ 0.132915] ************************************************************* [ 0.133029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134016] ** ** [ 0.135020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.136024] ** ** [ 0.137029] ** This means that this kernel is built to expose internal ** [ 0.138017] ** IOMMU data structures, which may compromise security on ** [ 0.139025] ** your system. ** [ 0.140020] ** ** [ 0.141018] ** If you see this message and you are not debugging the ** [ 0.142021] ** kernel, report this immediately to your vendor! ** [ 0.143021] ** ** [ 0.144029] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145019] ************************************************************* [ 0.147000] NET: Registered protocol family 16 [ 0.149539] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.155080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.159173] cpuidle: using governor menu [ 0.162099] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165790] PCI: Using configuration type 1 for base access [ 0.169160] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.180163] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.183030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187066] cryptd: max_cpu_qlen set to 1000 [ 0.189217] ACPI: Added _OSI(Module Device) [ 0.191017] ACPI: Added _OSI(Processor Device) [ 0.193018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.200798] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207546] ACPI: Interpreter enabled [ 0.208063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210015] ACPI: Using IOAPIC for interrupt routing [ 0.211000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214656] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.232205] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.235240] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239035] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249540] acpiphp: Slot [2] registered [ 0.251199] acpiphp: Slot [5] registered [ 0.253152] acpiphp: Slot [6] registered [ 0.255143] acpiphp: Slot [3] registered [ 0.257122] acpiphp: Slot [4] registered [ 0.259227] acpiphp: Slot [7] registered [ 0.261189] acpiphp: Slot [8] registered [ 0.263159] acpiphp: Slot [9] registered [ 0.264134] acpiphp: Slot [10] registered [ 0.266173] acpiphp: Slot [11] registered [ 0.268124] acpiphp: Slot [12] registered [ 0.269144] acpiphp: Slot [13] registered [ 0.271126] acpiphp: Slot [14] registered [ 0.273170] acpiphp: Slot [15] registered [ 0.275161] acpiphp: Slot [16] registered [ 0.276129] acpiphp: Slot [17] registered [ 0.278144] acpiphp: Slot [18] registered [ 0.279131] acpiphp: Slot [19] registered [ 0.281106] acpiphp: Slot [20] registered [ 0.283123] acpiphp: Slot [21] registered [ 0.285131] acpiphp: Slot [22] registered [ 0.286121] acpiphp: Slot [23] registered [ 0.288150] acpiphp: Slot [24] registered [ 0.289124] acpiphp: Slot [25] registered [ 0.291159] acpiphp: Slot [26] registered [ 0.293142] acpiphp: Slot [27] registered [ 0.294125] acpiphp: Slot [28] registered [ 0.296117] acpiphp: Slot [29] registered [ 0.297114] acpiphp: Slot [30] registered [ 0.299136] acpiphp: Slot [31] registered [ 0.301074] PCI host bridge to bus 0000:00 [ 0.302027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317056] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.319334] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323503] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327463] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.342524] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.349033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.351023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355777] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.358878] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.362064] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365991] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.372019] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383732] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.394612] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.420024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.428811] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.443019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.482285] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.486558] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.489551] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.492377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.494272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.499153] iommu: Default domain type: Passthrough [ 0.501577] SCSI subsystem initialized [ 0.503214] ACPI: bus type USB registered [ 0.505156] usbcore: registered new interface driver usbfs [ 0.507119] usbcore: registered new interface driver hub [ 0.509099] usbcore: registered new device driver usb [ 0.511302] pps_core: LinuxPPS API ver. 1 registered [ 0.514021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.517079] PTP clock support registered [ 0.520070] EDAC MC: Ver: 3.0.0 [ 0.522265] PCI: Using ACPI for IRQ routing [ 0.524080] NetLabel: Initializing [ 0.526028] NetLabel: domain hash size = 128 [ 0.528015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.530142] NetLabel: unlabeled traffic allowed by default [ 0.531146] vgaarb: loaded [ 0.533447] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.535015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.542612] clocksource: Switched to clocksource kvm-clock [ 0.659919] VFS: Disk quotas dquot_6.6.0 [ 0.661767] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.664815] *** VALIDATE ramfs *** [ 0.666314] *** VALIDATE hugetlbfs *** [ 0.667989] pnp: PnP ACPI init [ 0.670258] pnp: PnP ACPI: found 6 devices [ 0.692232] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.696690] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.700919] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.703988] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.707026] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.709893] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.713425] NET: Registered protocol family 2 [ 0.716288] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.722130] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.726138] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.732139] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.736641] TCP: Hash tables configured (established 65536 bind 65536) [ 0.740295] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.743677] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747054] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.751555] NET: Registered protocol family 1 [ 0.757703] RPC: Registered named UNIX socket transport module. [ 0.760308] RPC: Registered udp transport module. [ 0.761651] RPC: Registered tcp transport module. [ 0.763061] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.765674] NET: Registered protocol family 44 [ 0.767728] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.770057] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.772081] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.774743] PCI: CLS 0 bytes, default 64 [ 0.776942] Unpacking initramfs... [ 2.279313] debug: unmapping init [mem 0xffff895cbcc64000-0xffff895cbffcffff] [ 2.283435] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.285523] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.288256] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.821831] Initialise system trusted keyrings [ 2.824661] Key type blacklist registered [ 2.827199] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.839839] zbud: loaded [ 2.844830] *** VALIDATE nfs *** [ 2.846162] *** VALIDATE nfs4 *** [ 2.848280] pstore: using deflate compression [ 2.854515] Platform Keyring initialized [ 2.989838] NET: Registered protocol family 38 [ 2.992329] Key type asymmetric registered [ 2.994342] Asymmetric key parser 'x509' registered [ 2.996932] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.000978] io scheduler mq-deadline registered [ 3.003326] io scheduler kyber registered [ 3.005689] io scheduler bfq registered [ 3.008648] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.012802] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.016993] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.020641] ACPI: Power Button [PWRF] [ 3.026813] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.034617] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.045896] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.072698] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.104097] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.109393] Non-volatile memory driver v1.3 [ 3.111520] Linux agpgart interface v0.103 [ 3.155571] virtio_blk virtio1: [vda] 150616 512-byte logical blocks (77.1 MB/73.5 MiB) [ 3.159158] vda: detected capacity change from 0 to 77115392 [ 3.182508] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.186393] vdb: detected capacity change from 0 to 1073741824 [ 3.195051] libphy: Fixed MDIO Bus: probed [ 3.200990] usbcore: registered new interface driver usbserial_generic [ 3.204164] usbserial: USB Serial support registered for generic [ 3.207089] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.213792] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.215857] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.218900] mousedev: PS/2 mouse device common for all mice [ 3.222714] rtc_cmos 00:05: RTC can wake from S4 [ 3.226052] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.226944] rtc_cmos 00:05: registered as rtc0 [ 3.232543] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.234061] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.236724] intel_pstate: CPU model not supported [ 3.243437] hid: raw HID events driver (C) Jiri Kosina [ 3.246715] usbcore: registered new interface driver usbhid [ 3.246841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.249403] usbhid: USB HID core driver [ 3.249681] drop_monitor: Initializing network drop monitor service [ 3.256676] Initializing XFRM netlink socket [ 3.259335] NET: Registered protocol family 10 [ 3.262577] Segment Routing with IPv6 [ 3.264276] NET: Registered protocol family 17 [ 3.266851] mpls_gso: MPLS GSO support [ 3.273115] RAS: Correctable Errors collector initialized. [ 3.277542] AVX version of gcm_enc/dec engaged. [ 3.282783] AES CTR mode by8 optimization enabled [ 3.376230] sched_clock: Marking stable (3376091320, 0)->(4370675422, -994584102) [ 3.380333] registered taskstats version 1 [ 3.382153] Loading compiled-in X.509 certificates [ 3.384353] zswap: loaded using pool lzo/zbud [ 3.411732] Key type big_key registered [ 3.426267] Key type encrypted registered [ 3.428749] ima: No TPM chip found, activating TPM-bypass! [ 3.431198] ima: Allocated hash algorithm: sha1 [ 3.434863] ima: No architecture policies found [ 3.436790] evm: Initialising EVM extended attributes: [ 3.438631] evm: security.selinux [ 3.439793] evm: security.ima [ 3.441264] evm: security.capability [ 3.442767] evm: HMAC attrs: 0x1 [ 3.445445] rtc_cmos 00:05: setting system clock to 2026-10-02 17:59:29 UTC (1790963969) [ 3.452494] debug: unmapping init [mem 0xffffffff93603000-0xffffffff937fffff] [ 3.456400] debug: unmapping init [mem 0xffffffff92382000-0xffffffff92658fff] [ 3.466146] Write protecting the kernel read-only data: 28672k [ 3.470495] debug: unmapping init [mem 0xffffffff90a03000-0xffffffff90bfffff] [ 3.473464] debug: unmapping init [mem 0xffffffff91314000-0xffffffff913fffff] [ 3.509232] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.520511] systemd[1]: Detected virtualization kvm. [ 3.522158] systemd[1]: Detected architecture x86-64. [ 3.524134] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.550848] systemd[1]: No hostname configured. [ 3.553226] systemd[1]: Set hostname to . [ 3.556921] random: systemd: uninitialized urandom read (16 bytes read) [ 3.560703] systemd[1]: Initializing machine ID from random generator. [ 3.608728] random: ln: uninitialized urandom read (6 bytes read) [ 3.715458] random: systemd: uninitialized urandom read (16 bytes read) [ 3.718298] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.722909] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.730054] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.399300] device-mapper: uevent: version 1.0.3 [ 4.401945] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 5.088580] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.128647] virtio_net virtio0 ens2: renamed from eth0 [ 5.138709] scsi host0: ata_piix [ 5.151167] scsi host1: ata_piix [ 5.153403] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.159512] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.901562] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.948193] random: crng init done [ 9.950408] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.410054] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.560713] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.832592] SELinux: Disabled at runtime. [ 11.894286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.905256] systemd[1]: Detected virtualization kvm. [ 11.907480] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.414118] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.418663] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.424489] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.430302] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.434832] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.446159] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.455807] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ 12.628849] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.953621] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.254858] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.346479] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.387932] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.411815] EDAC sbridge: Ver: 1.1.2 [ 14.721093] Key type dns_resolver registered [ 15.026829] NFS: Registering the id_resolver key type [ 15.029269] Key type id_resolver registered [ 15.030991] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg427-client login: [ 84.363912] libcfs: loading out-of-tree module taints kernel. [ 84.500799] Key type ._llcrypt registered [ 84.506250] Key type .llcrypt registered [ 84.935709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 84.950608] alg: No test for adler32 (adler32-zlib) [ 86.636721] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 87.710940] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 89.503844] Key type lgssc registered [ 91.424461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.253693] hrtimer: interrupt took 9177553 ns [ 278.022500] Lustre: Mounted lustre-client - version 2.17.59_2_g3d9f037 [ 283.567499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 297.779756] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing check_logdir /tmp/testlogs/ [ 303.588579] Lustre: lustre-OST0000-osc-ffff895d04b60000: disconnect after 23s idle [ 303.722843] Lustre: DEBUG MARKER: oleg427-client.virtnet: executing yml_node [ 308.905318] Lustre: DEBUG MARKER: Client: 2.17.59.2 [ 312.168831] Lustre: DEBUG MARKER: MDS: 2.17.59.2 [ 314.708378] Lustre: DEBUG MARKER: OSS: 2.17.59.2 [ 316.287466] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Oct 2 14:04:41 EDT 2026 [ 329.767101] Lustre: Unmounted lustre-client [ 385.332974] Key type lgssc unregistered [ 385.587845] LNet: 9217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.600468] LNetError: 9217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.621989] LNet: Removed LNI 192.168.204.27@tcp [ 386.423166] Key type .llcrypt unregistered [ 386.432658] Key type ._llcrypt unregistered [ 416.904636] Key type ._llcrypt registered [ 416.906578] Key type .llcrypt registered [ 417.084429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 417.100685] alg: No test for adler32 (adler32-zlib) [ 417.981414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 418.056612] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 418.105289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 422.140995] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 422.223565] LNet: 10150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.235790] LNetError: 10150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.257048] LNet: Removed LNI 192.168.204.27@tcp [ 422.907596] Key type .llcrypt unregistered [ 422.911154] Key type ._llcrypt unregistered [ 428.495897] Key type ._llcrypt registered [ 428.498421] Key type .llcrypt registered [ 429.637691] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:06:34 (1790964394) [ 429.977215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 429.991973] alg: No test for adler32 (adler32-zlib) [ 430.890828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.971888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.001365] LNet: 10572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.036568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.121460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.141555] LNet: 10580:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 437.204439] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:06:41 (1790964401) [ 437.473883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.519612] LNet: 11166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.539207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.609458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 443.491027] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:06:48 (1790964408) [ 443.592669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.615946] LNet: 11753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.650064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.741663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 449.074065] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:06:54 (1790964414) [ 449.362041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.384708] LNet: 12352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.404262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.455202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 455.123991] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:06:59 (1790964419) [ 455.196973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.221461] LNet: 12939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 455.260086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.327695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 455.704796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 461.607645] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:07:06 (1790964426) [ 461.779281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.833412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.893375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 462.419952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 462.801467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 468.474738] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:07:13 (1790964433) [ 468.662625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.677865] LNet: 14147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.682843] LNet: 14147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 468.701608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.765117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 474.148287] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:07:18 (1790964438) [ 474.253551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.286444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.354644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 474.380533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 474.744926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 474.798901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 475.100286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 475.151328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 475.459404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 475.487781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 475.749897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 475.791850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 476.077362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 476.118538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 476.597328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 476.622816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 476.852735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 476.895892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 477.339533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 477.374716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 477.731547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 477.757042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 478.062436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 478.108720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 478.460634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 478.496579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 478.877517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 478.913626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 484.722865] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:07:29 (1790964449) [ 484.990835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.019273] LNet: 15429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.025285] LNet: 15429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 485.057755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.130717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 485.174679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 492.163249] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:07:36 (1790964456) [ 492.241440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.311619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.383428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 492.419868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 499.929842] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:07:44 (1790964464) [ 500.070997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.153616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.268906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 500.298698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 508.266251] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:07:52 (1790964472) [ 508.468937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.498378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.573894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 508.613552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 515.548912] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:07:59 (1790964479) [ 515.714637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.789033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.890381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 515.929594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 522.470593] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:08:07 (1790964487) [ 522.567175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.604185] LNet: 18394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.609033] LNet: 18394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 522.650649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.772608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 522.825580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 529.369567] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:08:14 (1790964494) [ 529.757846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.783319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.862570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.531604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 537.560890] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:08:21 (1790964501) [ 537.687314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.744664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.026431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 538.354338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.527976] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:08:31 (1790964511) [ 547.694703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.805345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.949832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.329058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.622564] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:08:41 (1790964521) [ 557.731620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.851578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.677871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.016504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 565.509116] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:08:50 (1790964530) [ 565.553565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.583449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.661773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.701762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 571.914777] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:08:56 (1790964536) [ 572.027505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.078339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.155501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.239341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 577.437605] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:09:02 (1790964542) [ 577.722558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.777927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.857605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 583.692569] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:09:08 (1790964548) [ 583.898406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.952712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.016756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 590.748747] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:09:15 (1790964555) [ 591.019797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.054834] LNet: 23908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.061606] LNet: 23908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 591.102926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.274456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 591.735267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 599.357653] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:09:24 (1790964564) [ 599.475857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.538675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.605059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 599.676913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 606.324843] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:09:31 (1790964571) [ 606.787822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.839840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.936472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 606.963505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 612.973603] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:09:37 (1790964577) [ 613.148147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.280716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.379776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 613.415240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 619.641499] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:09:44 (1790964584) [ 619.750603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.793723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.862825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 619.891519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 626.666589] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:09:51 (1790964591) [ 626.858655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.914518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.987546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 627.102137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 627.150051] LNetError: 26899:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 627.225857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 627.288572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 633.385659] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:09:58 (1790964598) [ 633.644549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.689686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.800213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hk0Er8EynL [ 633.932092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hk0Er8EynL [ 634.007876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hk0Er8EynL [ 640.342279] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:10:04 (1790964604) [ 640.550381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.589433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.637428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 640.671034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 640.719130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 646.820758] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:10:11 (1790964611) [ 646.954843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.034475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.116582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.151947] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 648.510943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 649.687194] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 651.583606] Key type lgssc registered [ 652.830563] Lustre: Echo OBD driver; http://www.lustre.org/ [ 656.206047] LNetError: 28684:0:(lib-move.c:2173:lnet_handle_find_routed_path()) peer 192.168.204.27@tcp99 has no available nets [ 658.966777] Key type lgssc unregistered [ 659.377429] LNetError: 29826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.415283] LNet: Removed LNI 192.168.204.27@tcp [ 659.958278] Key type .llcrypt unregistered [ 659.961126] Key type ._llcrypt unregistered [ 666.891907] Key type ._llcrypt registered [ 666.894466] Key type .llcrypt registered [ 668.351778] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:10:32 (1790964632) [ 668.927577] Key type .llcrypt unregistered [ 668.930584] Key type ._llcrypt unregistered [ 675.696227] Key type ._llcrypt registered [ 675.699514] Key type .llcrypt registered [ 675.958783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.977501] alg: No test for adler32 (adler32-zlib) [ 676.965211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.017345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 677.031553] LNet: 30759:0:(router.c:1482:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 677.038828] LNet: 30759:0:(router.c:1504:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 677.044718] LNet: 30759:0:(router.c:1526:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 677.055686] LNet: Message forwarding will be enabled in 2s [ 680.160286] LNet: Message forwarding enabled [ 680.535648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 680.561617] LNet: Message forwarding will stop in 2s to 6s [ 684.270786] LNet: Message forwarding disabled [ 685.254463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.295604] LNet: 30857:0:(router.c:1482:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 685.313174] LNet: 30857:0:(router.c:1504:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 685.322577] LNet: 30857:0:(router.c:1526:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 685.352758] LNet: Message forwarding will be enabled in 2s [ 685.674579] LNet: 30870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.701832] Key type .llcrypt unregistered [ 686.704873] Key type ._llcrypt unregistered [ 700.794811] Key type ._llcrypt registered [ 700.798084] Key type .llcrypt registered [ 702.669648] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:11:06 (1790964666) [ 703.083448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 703.128886] alg: No test for adler32 (adler32-zlib) [ 704.216948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.284802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 711.654516] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:11:16 (1790964676) [ 712.003818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.052422] LNet: 32291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.075159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.178712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 712.231899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 712.285421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 712.333858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 712.395669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 712.435775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 712.493957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 712.571130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 712.617775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 712.649215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 712.682945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 712.714888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 712.746958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 712.806062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 712.849662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 712.881827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 712.976858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 713.027225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 713.096842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 713.141454] LNetError: 32335:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 721.461709] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:11:26 (1790964686) [ 721.763144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.799357] LNet: 32914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.819678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.908975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 731.602487] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:11:35 (1790964695) [ 733.636978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg427-server [ 735.623571] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:11:40 (1790964700) [ 735.853052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.926837] LNet: 33857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.986980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.119157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.157799] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 737.373771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[28-32]@tcp [ 737.445305] LNet: 33995:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 737.579198] LNetError: 33866:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 737.622562] LNetError: 33866:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.28@tcp has gone from up to down [ 746.447185] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:11:50 (1790964710) [ 748.321918] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg427-server [ 750.170116] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:11:54 (1790964714) [ 750.273462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.304786] LNetError: 33866:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.29@tcp has gone from up to down [ 750.314339] LNetError: 33866:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 750.320128] LNet: 34940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.323928] LNetError: 34940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 750.356892] LNet: Removed LNI 192.168.204.27@tcp [ 750.408757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.501826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.530321] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 751.355305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[28-33/2]@tcp [ 751.394255] LNet: 35064:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 751.405711] LNet: 35064:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 751.448756] LNetError: 34951:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.28@tcp is being used as a gateway but routing feature is not turned on [ 751.453042] LNetError: 34958:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.32@tcp has gone from up to down [ 751.460114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[28-33/2]@tcp [ 751.478982] LNetError: 34951:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 751.537532] LNetError: 34958:0:(lib-lnet.h:1283:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 758.880742] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:12:03 (1790964723) [ 759.032417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.085469] LNet: 35659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.100215] LNetError: 35659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.125481] LNet: Removed LNI 192.168.204.27@tcp [ 759.165361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.308534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 759.384603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.443349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.531551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.591274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.642228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.708252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 759.793281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.860693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.930646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.037233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.091558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.159523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 760.246515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.286681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.365978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.422438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.459214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.495159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 760.559098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.594716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.653775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.798972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.865169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.917614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 761.026690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.103714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.232687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 761.421102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.507591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.593790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 761.713403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.766689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.859624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 770.178084] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:12:14 (1790964734) [ 772.557558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg427-server [ 774.926959] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:12:19 (1790964739) [ 777.845216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg427-server [ 780.269765] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:12:24 (1790964744) [ 780.696906] LNet: 37088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.711344] LNet: 37088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 782.306890] Key type .llcrypt unregistered [ 782.313595] Key type ._llcrypt unregistered [ 792.359346] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 792.464664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 792.489187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 792.506490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.122580] Key type ._llcrypt registered [ 794.124500] Key type .llcrypt registered [ 794.355920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.385880] alg: No test for adler32 (adler32-zlib) [ 795.335991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.383446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.402280] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 796.204733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.225061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.634458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 799.656872] LNet: Removed LNI 10.1.2.3@tcp [ 799.826665] LNet: 37801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.830960] LNetError: 37801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.843301] LNet: Removed LNI 192.168.204.27@tcp [ 800.446712] Key type .llcrypt unregistered [ 800.450090] Key type ._llcrypt unregistered [ 813.263991] Key type ._llcrypt registered [ 813.270538] Key type .llcrypt registered [ 814.686304] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:12:59 (1790964779) [ 815.161461] Key type .llcrypt unregistered [ 815.166213] Key type ._llcrypt unregistered [ 821.564522] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.758952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.762283] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.765229] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.332865] Key type ._llcrypt registered [ 824.338223] Key type .llcrypt registered [ 824.590547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.615627] alg: No test for adler32 (adler32-zlib) [ 825.567627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.629699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 825.657404] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 826.334145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 826.354982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.968064] LNet: 39304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.986794] LNetError: 39304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 828.992793] LNet: Removed LNI 10.1.2.3@tcp [ 829.005386] LNet: Removed LNI 192.168.204.27@tcp [ 829.529676] Key type .llcrypt unregistered [ 829.532800] Key type ._llcrypt unregistered [ 841.782288] Key type ._llcrypt registered [ 841.784792] Key type .llcrypt registered [ 843.363930] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:13:27 (1790964807) [ 844.026318] Key type .llcrypt unregistered [ 844.030070] Key type ._llcrypt unregistered [ 851.025537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 851.211941] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 851.229795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 851.232945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 853.774266] Key type ._llcrypt registered [ 853.779721] Key type .llcrypt registered [ 854.179916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.208434] alg: No test for adler32 (adler32-zlib) [ 855.396681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.572665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 855.671163] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 858.232882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 858.258952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.702397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 860.722826] LNet: Removed LNI 10.1.2.3@tcp [ 860.739986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 860.773192] LNet: Removed LNI 10.1.2.31@tcp [ 860.796899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 860.820926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.906842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 861.956252] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 863.040654] LNet: 41236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.080516] LNetError: 41236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 863.088931] LNet: Removed LNI 10.1.2.31@tcp [ 863.106451] LNet: Removed LNI 10.1.2.3@tcp [ 863.655221] Key type .llcrypt unregistered [ 863.659101] Key type ._llcrypt unregistered [ 878.312655] Key type ._llcrypt registered [ 878.314217] Key type .llcrypt registered [ 880.066654] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:14:04 (1790964844) [ 880.567711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 880.585499] alg: No test for adler32 (adler32-zlib) [ 881.647263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.779586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.840202] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 883.133150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.179072] LNet: 42199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.187613] LNetError: 42199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.205436] LNet: Removed LNI 192.168.204.27@tcp [ 883.251679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.370477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.27@tcp [ 883.419197] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 884.956514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.27@tcp [ 885.030663] LNet: Removed LNI 192.168.204.27@tcp [ 885.163603] LNet: 42334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.932020] Key type .llcrypt unregistered [ 885.934984] Key type ._llcrypt unregistered [ 905.360492] Key type ._llcrypt registered [ 905.362231] Key type .llcrypt registered [ 907.876888] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:14:31 (1790964871) [ 910.897811] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg427-server [ 913.366795] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:14:37 (1790964877) [ 914.783453] Key type .llcrypt unregistered [ 914.792277] Key type ._llcrypt unregistered [ 924.443865] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 924.741238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 924.744200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 924.770040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 927.851204] Key type ._llcrypt registered [ 927.857807] Key type .llcrypt registered [ 928.209751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.228773] alg: No test for adler32 (adler32-zlib) [ 929.199928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.264349] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 930.366170] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 932.718377] LNet: 44192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.726262] LNetError: 44192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.733704] LNet: Removed LNI 192.168.204.27@tcp1 [ 932.752228] LNet: Removed LNI 10.1.2.3@tcp2 [ 933.851339] Key type .llcrypt unregistered [ 933.860161] Key type ._llcrypt unregistered [ 949.635739] Key type ._llcrypt registered [ 949.638436] Key type .llcrypt registered [ 951.204447] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:15:15 (1790964915) [ 951.692833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.715365] alg: No test for adler32 (adler32-zlib) [ 952.822737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.910683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.952166] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 954.215672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.256898] LNet: 45171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.265321] LNetError: 45171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.277132] LNet: Removed LNI 192.168.204.27@tcp [ 954.298866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.372190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 954.409443] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 962.659949] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:15:27 (1790964927) [ 962.970600] LNet: 45767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.989528] LNetError: 45767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.013164] LNet: Removed LNI 192.168.204.27@tcp [ 964.027498] Key type .llcrypt unregistered [ 964.030643] Key type ._llcrypt unregistered [ 971.499567] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 971.717255] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.722452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.737845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.274254] Key type ._llcrypt registered [ 974.278914] Key type .llcrypt registered [ 974.519780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.534518] alg: No test for adler32 (adler32-zlib) [ 975.553697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.624539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.668771] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 976.593079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 976.641455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.254615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.276435] LNet: 46513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.282375] LNetError: 46513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.287592] LNet: Removed LNI 10.1.2.3@tcp [ 979.297644] LNet: Removed LNI 192.168.204.27@tcp [ 979.317706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.372906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 979.392117] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 979.399119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 979.895107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.918841] LNet: 46540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.928245] LNetError: 46540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.936217] LNetError: 46540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 979.945398] LNet: Removed LNI 10.1.2.3@tcp [ 979.963526] LNet: Removed LNI 192.168.204.27@tcp [ 979.991783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.048370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 980.075789] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 980.083602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.690504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 980.715620] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 981.863721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.881752] LNet: 46702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.887289] LNetError: 46702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 981.892375] LNetError: 46702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 981.897459] LNet: Removed LNI 10.1.2.3@tcp [ 981.902310] LNet: Removed LNI 192.168.204.27@tcp [ 981.914894] LNet: Removed LNI 192.168.204.27@tcp2 [ 981.941887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.062337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 982.113994] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 982.121058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.128153] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 991.949174] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:15:56 (1790964956) [ 992.116387] LNet: 47303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.124391] LNetError: 47303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.130868] LNetError: 47303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 992.147285] LNet: Removed LNI 10.1.2.3@tcp [ 992.164129] LNet: Removed LNI 192.168.204.27@tcp [ 992.187139] LNet: Removed LNI 192.168.204.27@tcp2 [ 993.059958] Key type .llcrypt unregistered [ 993.065971] Key type ._llcrypt unregistered [ 1000.882924] Key type ._llcrypt registered [ 1000.887988] Key type .llcrypt registered [ 1001.274574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.300137] alg: No test for adler32 (adler32-zlib) [ 1002.287455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.337391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1002.365739] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1003.245724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.275948] LNet: 47703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.284793] LNetError: 47703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.313485] LNet: Removed LNI 192.168.204.27@tcp [ 1003.355955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.452716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1003.505841] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1004.206447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1004.245807] LNet: 47733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.257347] LNetError: 47733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1004.272879] LNet: Removed LNI 192.168.204.27@tcp [ 1004.309269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.417108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1004.455766] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1005.120694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.202484] LNet: Removed LNI 192.168.204.27@tcp [ 1005.235166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.345307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1005.406945] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1005.976403] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1006.188425] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1006.193646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1006.197831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1008.808610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.836818] LNet: 47843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.841068] LNet: 47843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1008.847250] LNetError: 47843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.852665] LNetError: 47843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1008.868398] LNet: Removed LNI 192.168.204.27@tcp [ 1008.905353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.979795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1009.029276] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1017.923324] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:16:22 (1790964982) [ 1018.135746] LNet: 48449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.152848] LNetError: 48449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.180678] LNet: Removed LNI 192.168.204.27@tcp [ 1018.913168] Key type .llcrypt unregistered [ 1018.918397] Key type ._llcrypt unregistered [ 1027.058606] Key type ._llcrypt registered [ 1027.062653] Key type .llcrypt registered [ 1027.399894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.413426] alg: No test for adler32 (adler32-zlib) [ 1028.381273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.443402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.486629] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1029.169818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.183738] LNet: 48849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.187602] LNetError: 48849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.201185] LNet: Removed LNI 192.168.204.27@tcp [ 1029.225814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.324454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1029.365287] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1036.075214] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:16:40 (1790965000) [ 1036.163774] LNet: 49441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.168477] LNetError: 49441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.188195] LNet: Removed LNI 192.168.204.27@tcp [ 1037.259259] Key type .llcrypt unregistered [ 1037.267806] Key type ._llcrypt unregistered [ 1045.017909] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1045.163573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1045.166186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1045.168850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1047.738450] Key type ._llcrypt registered [ 1047.743666] Key type .llcrypt registered [ 1048.081952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.111867] alg: No test for adler32 (adler32-zlib) [ 1049.063964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.151642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1049.181172] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1050.267995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.308611] LNet: 49895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.321482] LNetError: 49895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.337451] LNet: Removed LNI 192.168.204.27@tcp [ 1050.380675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.492951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1050.536709] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1058.673766] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:17:03 (1790965023) [ 1059.337579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.392564] LNet: 50503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.430571] LNetError: 50503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.456062] LNet: Removed LNI 192.168.204.27@tcp [ 1059.478581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.627262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1066.929445] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:17:11 (1790965031) [ 1067.135290] LNet: 51092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.084599] Key type .llcrypt unregistered [ 1068.094082] Key type ._llcrypt unregistered [ 1074.445559] Key type ._llcrypt registered [ 1074.447741] Key type .llcrypt registered [ 1074.715525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.733841] alg: No test for adler32 (adler32-zlib) [ 1075.711044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1075.795623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1075.855420] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1077.522712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.590905] LNet: 51493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.604450] LNetError: 51493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.641558] LNet: Removed LNI 192.168.204.27@tcp [ 1077.713154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.901800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1077.988841] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1088.138218] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:17:32 (1790965052) [ 1088.350841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1088.399848] LNet: 52088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.429678] LNetError: 52088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.493092] LNet: Removed LNI 192.168.204.27@tcp [ 1088.561724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.737311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1098.039414] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:17:42 (1790965062) [ 1100.528584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1102.684788] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:17:46 (1790965066) [ 1104.704433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1107.484946] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:17:51 (1790965071) [ 1109.593825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1111.927510] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:17:56 (1790965076) [ 1113.743874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1115.887834] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:18:00 (1790965080) [ 1118.071822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1120.434759] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:18:04 (1790965084) [ 1120.837079] LNet: 54147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.739694] Key type .llcrypt unregistered [ 1121.745318] Key type ._llcrypt unregistered [ 1130.262641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1130.529530] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1130.555930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1130.566855] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1133.279326] Key type ._llcrypt registered [ 1133.281780] Key type .llcrypt registered [ 1133.492369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.530484] alg: No test for adler32 (adler32-zlib) [ 1134.525461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.632679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1134.668830] LNet: Added LNI 192.168.204.27@tcp [8/512/0/180] [ 1134.678954] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1134.682687] LNet: 54469:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1134.866759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.901542] LNet: 54488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.913330] LNetError: 54488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1134.920788] LNet: Removed LNI 10.1.2.3@tcp [ 1134.923671] LNetError: 54488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.935488] LNet: Removed LNI 192.168.204.27@tcp [ 1134.960660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.050519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1135.074630] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1135.145702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1135.164989] LNet: 54512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.177559] LNetError: 54512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.196892] LNet: Removed LNI 192.168.204.27@tcp [ 1135.223192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.303450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1135.326505] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1135.333792] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.336295] LNet: 54520:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1135.437322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1135.498431] LNet: 54538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.509743] LNetError: 54538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1135.522865] LNet: Removed LNI 10.1.2.3@tcp [ 1135.542776] LNetError: 54538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.573921] LNet: Removed LNI 192.168.204.27@tcp [ 1135.612548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.697432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1135.769770] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1135.777280] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.789419] LNet: 54546:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1136.069123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1136.113266] LNet: 54572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.124426] LNetError: 54572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1136.135266] LNet: Removed LNI 10.1.2.3@tcp [ 1136.144524] LNetError: 54572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.167363] LNet: Removed LNI 192.168.204.27@tcp [ 1136.200529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.316179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1136.338313] LNet: Added LNI 192.168.204.27@tcp [8/256/0/182] [ 1136.343247] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1136.487725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1136.537575] LNet: 54600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.543661] LNetError: 54600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1136.553258] LNet: Removed LNI 10.1.2.3@tcp [ 1136.554912] LNetError: 54600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.591232] LNet: Removed LNI 192.168.204.27@tcp [ 1136.632677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.727930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1136.761755] LNet: Added LNI 192.168.204.27@tcp [8/256/0/182] [ 1136.768379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1136.770434] LNet: 54608:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1136.902769] LNet: Removed LNI 10.1.2.3@tcp [ 1136.945493] LNet: Removed LNI 192.168.204.27@tcp [ 1137.743754] Key type .llcrypt unregistered [ 1137.747024] Key type ._llcrypt unregistered [ 1151.935479] Key type ._llcrypt registered [ 1151.940792] Key type .llcrypt registered [ 1153.603362] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:18:37 (1790965117) [ 1155.300369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1157.165594] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:18:41 (1790965121) [ 1157.621242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.650755] alg: No test for adler32 (adler32-zlib) [ 1158.598100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1158.662407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1158.703109] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1159.628168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.27 [ 1159.656783] LNet: 55886:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1159.668967] LNet: Removed LNI 192.168.204.27@tcp [ 1166.875168] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:18:51 (1790965131) [ 1167.000251] LNet: 56457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.020234] LNetError: 56457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.053962] LNet: Removed LNI 192.168.204.27@tcp [ 1168.109385] Key type .llcrypt unregistered [ 1168.113941] Key type ._llcrypt unregistered [ 1176.341933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1176.652576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1179.684669] Key type ._llcrypt registered [ 1179.687538] Key type .llcrypt registered [ 1179.936874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.970851] alg: No test for adler32 (adler32-zlib) [ 1181.605875] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1189.499602] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:19:14 (1790965154) [ 1190.088489] LNet: 57387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.098791] LNetError: 57387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1190.115167] LNet: Removed LNI 10.1.2.3@tcp [ 1190.810598] Key type .llcrypt unregistered [ 1190.814159] Key type ._llcrypt unregistered [ 1198.465125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1198.890503] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1198.904143] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1198.916138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1201.848842] Key type ._llcrypt registered [ 1201.853972] Key type .llcrypt registered [ 1202.142792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.169755] alg: No test for adler32 (adler32-zlib) [ 1203.262357] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1210.576184] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:19:34 (1790965174) [ 1212.832838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1214.640124] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:19:39 (1790965179) [ 1214.862111] LNet: 58579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.881655] LNetError: 58579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1214.930955] LNet: Removed LNI 10.1.2.3@tcp [ 1216.074554] Key type .llcrypt unregistered [ 1216.076269] Key type ._llcrypt unregistered [ 1225.064197] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1225.559910] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1225.566984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1225.575721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1228.471576] Key type ._llcrypt registered [ 1228.474634] Key type .llcrypt registered [ 1228.626717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1228.654909] alg: No test for adler32 (adler32-zlib) [ 1229.746488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1229.850581] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1241.075292] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:20:05 (1790965205) [ 1241.771538] LNet: 59826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.787031] LNetError: 59826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1241.799885] LNet: Removed LNI 10.1.2.3@tcp [ 1242.967575] Key type .llcrypt unregistered [ 1242.979901] Key type ._llcrypt unregistered [ 1250.840892] Key type ._llcrypt registered [ 1250.842699] Key type .llcrypt registered [ 1251.200786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.231991] alg: No test for adler32 (adler32-zlib) [ 1252.208320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.268564] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1261.370591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1262.322795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_interrupt -r 1 [ 1262.369838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_interrupt -r 1 [ 1262.438286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1262.472927] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1262.563165] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 1262.821255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.888123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1262.961895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.005248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.068432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1263.174535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.263135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.364794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.431426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.458909] LNet: 60275:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.487653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.523756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.575646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.584181] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1263.657764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.724446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.786794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.829192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.637472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_dropped -r 1 [ 1264.675230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_dropped -r 1 [ 1264.713634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1264.741142] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1265.059925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1265.088255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1265.122841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.174284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1265.229153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1265.271164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.319203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.376344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.412338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.425866] LNet: 60386:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.444719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.492754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.524970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.564089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.616169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.667632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.704329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.353775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_aborted -r 1 [ 1266.426849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_aborted -r 1 [ 1266.482758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1266.861558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.902747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1266.942953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.979739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1267.026658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1267.080475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1267.135313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1267.195082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1267.299934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1267.329398] LNet: 60497:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1267.361496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1267.400080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1267.428945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.479968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1267.530028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1267.571754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1267.628977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1268.295856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_no_route -r 1 [ 1268.356974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_no_route -r 1 [ 1268.456406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1268.487912] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1268.495210] LNet: Skipped 2 previous similar messages [ 1268.787958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.847633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.902476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.948572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.982568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1269.033991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.073906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.127507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.171171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.220097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.253986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.278784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.303255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.329278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.354431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1269.400855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1270.061742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_timeout -r 1 [ 1270.142135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_timeout -r 1 [ 1270.200693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1270.524670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.560650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1270.600960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.627580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.679863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1270.718604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.753919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.784992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.836173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.870705] LNet: 60720:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1270.877719] LNet: 60720:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1270.908225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.938964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.969843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1271.019920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1271.089336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1271.158414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1271.237991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1271.890905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_error -r 1 [ 1271.961994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_error -r 1 [ 1272.028677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1272.366076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.472749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.524617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.563854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1272.617582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1272.678826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1272.745811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1272.801916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1272.866832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1272.896946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1272.936831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1272.986485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.035869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.082875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.157577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.271829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1273.690396] LNet: 60872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.697942] LNetError: 60872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.725094] LNet: Removed LNI 192.168.204.27@tcp [ 1274.172878] Key type .llcrypt unregistered [ 1274.175623] Key type ._llcrypt unregistered [ 1295.347778] Key type ._llcrypt registered [ 1295.352420] Key type .llcrypt registered [ 1297.159545] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:21:01 (1790965261) [ 1298.360573] Key type .llcrypt unregistered [ 1298.363343] Key type ._llcrypt unregistered [ 1306.508744] Key type ._llcrypt registered [ 1306.512827] Key type .llcrypt registered [ 1306.718319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1306.754130] alg: No test for adler32 (adler32-zlib) [ 1307.728775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1307.768233] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1315.484427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1328.156282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.238031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.287191] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1329.669942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1330.271775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_interrupt -r 1 [ 1330.343295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_interrupt -r 1 [ 1330.412805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_interrupt -r 1 [ 1330.448757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_interrupt -r 1 [ 1330.548942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_interrupt -r 1 [ 1330.628325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_interrupt -r 1 [ 1330.703821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_interrupt -r 1 [ 1330.771749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_interrupt -r 1 [ 1330.826628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1330.853694] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1330.862801] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp1 [ 1331.171388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.202636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.240565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.274809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.300651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1331.342814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.379622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.410880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.444717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.464260] LNet: 62361:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.484588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1331.518989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1331.558230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1331.590901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.627858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.662668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.688497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1332.437677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_dropped -r 1 [ 1332.476834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_dropped -r 1 [ 1332.529610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_dropped -r 1 [ 1332.596211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_dropped -r 1 [ 1332.672690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_dropped -r 1 [ 1332.759700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_dropped -r 1 [ 1332.846810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_dropped -r 1 [ 1332.937335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_dropped -r 1 [ 1333.081861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1333.139582] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1333.154717] LNet: Skipped 2 previous similar messages [ 1333.471877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1333.520812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1333.559424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1333.614781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1333.647864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1333.691885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1333.725853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1333.779943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1333.855536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1333.904075] LNet: 62484:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1333.941586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.029071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.111437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.171043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.179959] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1334.203662] LNet: Skipped 3 previous similar messages [ 1334.223492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.277521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.324937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1335.054687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_aborted -r 1 [ 1335.115995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_aborted -r 1 [ 1335.184690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_aborted -r 1 [ 1335.236538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_aborted -r 1 [ 1335.298434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_aborted -r 1 [ 1335.341662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_aborted -r 1 [ 1335.396036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_aborted -r 1 [ 1335.456226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_aborted -r 1 [ 1335.526533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1335.847633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.876276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1335.920654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.963811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.033932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1336.091241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.153939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1336.188493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.235281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.265940] LNet: 62608:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.290367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.367944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.397341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.459445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.510513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.569564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.629649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1337.416759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_no_route -r 1 [ 1337.448807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_no_route -r 1 [ 1337.496593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_no_route -r 1 [ 1337.542989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_no_route -r 1 [ 1337.593846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_no_route -r 1 [ 1337.671694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_no_route -r 1 [ 1337.744795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_no_route -r 1 [ 1337.814654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_no_route -r 1 [ 1337.876757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1337.946284] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1337.963064] LNet: Skipped 3 previous similar messages [ 1338.375891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.430408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.477834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.532593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.590805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1338.681558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.746605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.810496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.902765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.940115] LNet: 62731:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.975297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.045559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.098685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.154501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.207839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.257372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.292119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1340.130218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_timeout -r 1 [ 1340.198552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_timeout -r 1 [ 1340.246877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_timeout -r 1 [ 1340.291803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_timeout -r 1 [ 1340.319937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_timeout -r 1 [ 1340.357803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_timeout -r 1 [ 1340.388644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_timeout -r 1 [ 1340.418420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_timeout -r 1 [ 1340.456346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1340.689839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.747485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.788349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.834917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.859857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1340.884919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.928838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.005270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.051705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.117643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.176260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.260533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.313698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.398209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.489217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.579282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1342.424463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_error -r 1 [ 1342.465747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_error -r 1 [ 1342.498591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_error -r 1 [ 1342.549691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_error -r 1 [ 1342.614640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_error -r 1 [ 1342.677716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_error -r 1 [ 1342.723789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_error -r 1 [ 1342.801567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_error -r 1 [ 1342.876730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1342.920355] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1342.928225] LNet: Skipped 7 previous similar messages [ 1343.296851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1343.356148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1343.418888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1343.475859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.517269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1343.558529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.613581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.661039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.705492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.727799] LNet: 62978:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.734766] LNet: 62978:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1343.759882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.790662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.823979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.873564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.904471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.935298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.970293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1350.722918] LNet: 63054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.731572] LNetError: 63054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.742824] LNet: Removed LNI 192.168.204.27@tcp [ 1351.792795] LNet: Removed LNI 192.168.204.27@tcp1 [ 1352.515744] Key type .llcrypt unregistered [ 1352.517960] Key type ._llcrypt unregistered [ 1374.790228] Key type ._llcrypt registered [ 1374.796569] Key type .llcrypt registered [ 1376.848488] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:22:20 (1790965340) [ 1377.765682] Key type .llcrypt unregistered [ 1377.769526] Key type ._llcrypt unregistered [ 1384.897221] Key type ._llcrypt registered [ 1384.902115] Key type .llcrypt registered [ 1385.133797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.145709] alg: No test for adler32 (adler32-zlib) [ 1386.142733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1386.179667] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1394.333734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1395.287820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_dropped -r 1 [ 1395.351226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_dropped -r 1 [ 1395.449609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1395.473392] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1395.792376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1395.829803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1395.871530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1395.911568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1395.959641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1396.002753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1396.033203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1396.069316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1396.109196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1396.125984] LNet: 64328:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.144166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1396.183695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1396.223221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.282757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.347521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.383476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1396.417052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1397.040134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_error -r 1 [ 1397.076168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_error -r 1 [ 1397.112703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1397.131770] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1397.390079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1397.424996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1397.460329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1397.494347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1397.529964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1397.582586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1397.646843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1397.714478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1397.761916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1397.789772] LNet: 64439:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1397.822762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1397.908214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1397.950642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1397.991245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1398.026635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.065758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.107945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1398.841156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_timeout -r 1 [ 1398.888717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_timeout -r 1 [ 1398.958264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1398.999794] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1399.201552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1399.234794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1399.270771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1399.324305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1399.398303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1399.442529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1399.483328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1399.528665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1399.575841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1399.603287] LNet: 64551:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1399.627363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1399.689990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1399.749725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.801029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1399.858977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1399.912688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.974501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1400.427739] LNet: 64592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.442368] LNetError: 64592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.473099] LNet: Removed LNI 192.168.204.27@tcp [ 1401.063603] Key type .llcrypt unregistered [ 1401.065195] Key type ._llcrypt unregistered [ 1419.499610] Key type ._llcrypt registered [ 1419.501820] Key type .llcrypt registered [ 1420.922369] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:23:05 (1790965385) [ 1422.197080] Key type .llcrypt unregistered [ 1422.202756] Key type ._llcrypt unregistered [ 1430.225707] Key type ._llcrypt registered [ 1430.228332] Key type .llcrypt registered [ 1430.648498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.690265] alg: No test for adler32 (adler32-zlib) [ 1431.835691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1431.888364] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1440.506672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1450.227822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.321280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1450.356281] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1451.371223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1452.392070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_dropped -r 1 [ 1452.459577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e remote_dropped -r 1 [ 1452.520706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_dropped -r 1 [ 1452.595440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e remote_dropped -r 1 [ 1452.642571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e remote_dropped -r 1 [ 1452.705777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e remote_dropped -r 1 [ 1452.777687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e remote_dropped -r 1 [ 1452.821732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e remote_dropped -r 1 [ 1452.854297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1452.893610] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1452.915389] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.127@tcp1 [ 1453.183895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1453.231693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1453.275636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1453.305150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1453.360543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1453.400720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1453.455698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1453.518782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1453.582546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1453.621925] LNet: 66078:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1453.659552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1453.695838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.736688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.791886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.861965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.906771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.973778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1454.088567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1454.963807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_error -r 1 [ 1455.011636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e remote_error -r 1 [ 1455.102256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_error -r 1 [ 1455.182783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e remote_error -r 1 [ 1455.257300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e remote_error -r 1 [ 1455.319538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e remote_error -r 1 [ 1455.382549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e remote_error -r 1 [ 1455.468055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e remote_error -r 1 [ 1455.507994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1455.536724] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1455.543539] LNet: Skipped 2 previous similar messages [ 1455.828242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1455.887818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1455.925837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.982463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1456.028244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1456.066549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1456.094129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1456.146779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1456.207647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1456.228257] LNet: 66204:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1456.279836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1456.376827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1456.438652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.509792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1456.581683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1456.628495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1456.686208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1456.796807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1457.510476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_timeout -r 1 [ 1457.654200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e remote_timeout -r 1 [ 1457.744226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_timeout -r 1 [ 1457.833584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e remote_timeout -r 1 [ 1457.922437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e remote_timeout -r 1 [ 1458.002668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e remote_timeout -r 1 [ 1458.070295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e remote_timeout -r 1 [ 1458.144325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e remote_timeout -r 1 [ 1458.198349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.127@tcp [ 1458.241607] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1458.248650] LNet: Skipped 1 previous similar message [ 1458.667737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1458.705376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1458.752564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1458.804969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1458.861501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1458.897091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1458.939828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1459.004457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1459.070686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1459.113588] LNet: 66329:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1459.163588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1459.216478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1459.261838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1459.309177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1459.339734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1459.364387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1459.397295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1459.427244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1464.729610] LNet: 66407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.752523] LNetError: 66407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.759723] LNet: Removed LNI 192.168.204.27@tcp [ 1464.776107] LNet: Removed LNI 192.168.204.27@tcp1 [ 1465.488408] Key type .llcrypt unregistered [ 1465.495675] Key type ._llcrypt unregistered [ 1485.258231] Key type ._llcrypt registered [ 1485.261061] Key type .llcrypt registered [ 1486.888280] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:24:11 (1790965451) [ 1489.236588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1491.817893] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:24:15 (1790965455) [ 1493.072425] Key type .llcrypt unregistered [ 1493.076931] Key type ._llcrypt unregistered [ 1501.705319] Key type ._llcrypt registered [ 1501.708386] Key type .llcrypt registered [ 1502.130535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1502.146570] alg: No test for adler32 (adler32-zlib) [ 1503.121571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1503.147348] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1512.301255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1513.071852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e network_timeout -r 1 [ 1513.125952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e network_timeout -r 1 [ 1513.161868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1513.177508] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1513.393260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1513.420686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1513.440333] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 1513.459441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1513.491518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1513.521902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1513.553596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1513.589388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1513.630287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.671735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1513.707394] LNet: 67971:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1513.739878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1513.789742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1513.868278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1513.930769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1514.000466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1514.046424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1514.376225] LNet: 68010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.384976] LNetError: 68010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.412593] LNet: Removed LNI 192.168.204.27@tcp [ 1515.012702] Key type .llcrypt unregistered [ 1515.020859] Key type ._llcrypt unregistered [ 1532.965606] Key type ._llcrypt registered [ 1532.970915] Key type .llcrypt registered [ 1533.353492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1533.398943] alg: No test for adler32 (adler32-zlib) [ 1534.300210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1534.330494] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1542.928280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1554.906792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.058763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.110686] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1557.017244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1558.181373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e network_timeout -r 1 [ 1558.225838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e network_timeout -r 1 [ 1558.279833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e network_timeout -r 1 [ 1558.334982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e network_timeout -r 1 [ 1558.436995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e network_timeout -r 1 [ 1558.519376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e network_timeout -r 1 [ 1558.590892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e network_timeout -r 1 [ 1558.665586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e network_timeout -r 1 [ 1558.709337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 1558.727978] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1558.944722] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp1 [ 1558.949851] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.127@tcp1 [ 1559.071293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1559.146186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1559.198110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1559.273059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1559.317761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1559.374727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1559.415904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1559.511599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1559.586435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1559.623335] LNet: 68903:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1559.689289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1559.769301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1559.846566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1559.910386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1559.980800] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1560.059864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.142719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1566.017146] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1566.032599] LNet: Skipped 1 previous similar message [ 1566.752586] LNet: 68977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.772558] LNetError: 68977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.799057] LNet: Removed LNI 192.168.204.27@tcp [ 1566.854386] LNet: Removed LNI 192.168.204.27@tcp1 [ 1567.686718] Key type .llcrypt unregistered [ 1567.688374] Key type ._llcrypt unregistered [ 1586.871485] Key type ._llcrypt registered [ 1586.874423] Key type .llcrypt registered [ 1588.447734] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:25:52 (1790965552) [ 1588.765944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.785265] alg: No test for adler32 (adler32-zlib) [ 1589.706936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.805831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.858498] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1591.084420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1591.111125] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1592.088489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 1592.159521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1592.198850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1592.237504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1592.266923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1592.297632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1592.348783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.27@tcp [ 1592.386638] LNetError: 70103:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.27@tcp to recovery [ 1592.863588] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 1592.864407] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1594.912807] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1595.921155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1599.008254] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1599.465235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1599.503695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1599.579490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.602663] LNet: 70210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.610840] LNetError: 70210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.617412] LNet: Removed LNI 192.168.204.27@tcp [ 1599.625741] LNet: Removed LNI 192.168.204.27@tcp1 [ 1599.680754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.730598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.765298] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1600.393709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1600.406703] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1601.215449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 1601.276556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1601.355686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1601.428953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1601.480565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1601.523363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1601.567310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1601.619523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.27@tcp [ 1601.641662] LNetError: 70482:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.27@tcp to recovery [ 1601.760478] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1604.012777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1607.904684] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1607.912501] LNet: Skipped 1 previous similar message [ 1613.247950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1613.327356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1613.396187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.667817] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:26:24 (1790965584) [ 1620.814034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.844253] LNet: 71206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1620.851259] LNetError: 71206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1620.863034] LNetError: 71206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1620.871074] LNet: Removed LNI 192.168.204.27@tcp [ 1620.891797] LNet: Removed LNI 192.168.204.27@tcp1 [ 1620.928755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.003853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.052768] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1622.032455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1622.065268] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1623.232894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 1623.346802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1623.377666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1623.423688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1623.473991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1623.548981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1623.624446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.27@tcp --health 0 [ 1623.826038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1624.035719] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.27@tcp [ 1624.035846] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1624.058226] LNet: Skipped 1 previous similar message [ 1638.903517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1639.156048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1639.225239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1639.260963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.27@tcp --health 500 [ 1639.454036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1646.243934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1646.312129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.331639] LNet: 71692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.342132] LNetError: 71692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.355121] LNetError: 71692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1646.366245] LNet: Removed LNI 192.168.204.27@tcp [ 1646.384614] LNet: Removed LNI 192.168.204.27@tcp1 [ 1646.432860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.518787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.555557] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1647.373697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1647.411722] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1648.638911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 1648.693978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.836767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1648.892645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1648.947441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1649.003716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1649.060891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1649.108839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.27@tcp --health 0 [ 1650.591907] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1650.609645] LNet: Skipped 3 previous similar messages [ 1651.611678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1661.178983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1661.212806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1661.264624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1671.584845] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:27:15 (1790965635) [ 1672.092514] LNet: 72750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.096942] LNetError: 72750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.101290] LNetError: 72750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1672.104755] LNet: Removed LNI 192.168.204.27@tcp [ 1672.119508] LNet: Removed LNI 192.168.204.27@tcp1 [ 1673.197872] Key type .llcrypt unregistered [ 1673.204747] Key type ._llcrypt unregistered [ 1681.663508] Key type ._llcrypt registered [ 1681.667875] Key type .llcrypt registered [ 1681.944634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1681.974782] alg: No test for adler32 (adler32-zlib) [ 1682.975529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1683.014792] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1695.075950] Lustre: DEBUG MARKER: Initial discovery [ 1695.179725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.127@tcp [ 1699.401994] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1699.471532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1699.520804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.127@tcp [ 1699.562246] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1699.604389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1699.661466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.127@tcp -r 1 -m GET -l 3 [ 1699.724299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.127@tcp -r 1 -m GET -e local_error [ 1699.786249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.127@tcp -r 1 -m PUT -l 6 [ 1700.459549] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 1701.870675] Lustre: DEBUG MARKER: Force oleg427-client.virtnet to discover 192.168.204.127@tcp (in background) [ 1701.981908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.127@tcp [ 1704.275895] Lustre: DEBUG MARKER: Force oleg427-server to discover 192.168.204.27@tcp [ 1705.055408] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1707.573591] Lustre: DEBUG MARKER: Wait for 73514 [ 1708.523407] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 1709.889090] Lustre: DEBUG MARKER: Finished wait on 73514 [ 1709.963392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1710.029336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1710.506508] LNet: 73981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.518653] LNetError: 73981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.557182] LNet: Removed LNI 192.168.204.27@tcp [ 1711.080912] Key type .llcrypt unregistered [ 1711.085639] Key type ._llcrypt unregistered [ 1729.848030] Key type ._llcrypt registered [ 1729.851825] Key type .llcrypt registered [ 1731.309457] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:28:15 (1790965695) [ 1731.990372] Key type .llcrypt unregistered [ 1731.995204] Key type ._llcrypt unregistered [ 1739.868111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1740.082510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1740.092571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1740.100446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1743.051414] Key type ._llcrypt registered [ 1743.063420] Key type .llcrypt registered [ 1743.333228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.372394] alg: No test for adler32 (adler32-zlib) [ 1744.352477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.434754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.468065] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1745.712653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1745.768290] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1757.449365] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:28:41 (1790965721) [ 1757.654667] LNet: 76072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.672447] LNetError: 76072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1757.683168] LNet: Removed LNI 10.1.2.3@tcp [ 1757.726038] LNet: Removed LNI 192.168.204.27@tcp [ 1758.457967] Key type .llcrypt unregistered [ 1758.462921] Key type ._llcrypt unregistered [ 1766.955863] Key type ._llcrypt registered [ 1766.957449] Key type .llcrypt registered [ 1767.117343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1767.136896] alg: No test for adler32 (adler32-zlib) [ 1768.148548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.261171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.302790] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1769.302492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1769.329409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1770.551773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.27@tcp [ 1771.619831] LNetError: Unexpected error -22 connecting to 192.168.204.27@tcp at host 192.168.204.27:988 [ 1777.332953] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:29:01 (1790965741) [ 1777.495982] LNet: 77207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.508374] LNetError: 77207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1777.526738] LNet: Removed LNI 10.1.2.3@tcp [ 1777.562615] LNet: Removed LNI 192.168.204.27@tcp [ 1778.388636] Key type .llcrypt unregistered [ 1778.393973] Key type ._llcrypt unregistered [ 1783.822358] Key type ._llcrypt registered [ 1783.825786] Key type .llcrypt registered [ 1784.030621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1784.049401] alg: No test for adler32 (adler32-zlib) [ 1784.986995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.044713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1785.065154] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1785.708806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1785.729506] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 1786.577153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.27@tcp1 --nid 192.168.204.27@tcp2 [ 1791.046576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp1 192.168.204.27@tcp2 [ 1791.064572] LNetError: 78052:0:(lib-move.c:2116:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.27@tcp1 to 192.168.204.27@tcp2 - Route table may be misconfigured [ 1791.087635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp2 192.168.204.27@tcp1 [ 1798.329731] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:29:23 (1790965763) [ 1798.720854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.745690] LNet: 78633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.752827] LNetError: 78633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.775338] LNet: Removed LNI 192.168.204.27@tcp1 [ 1799.799285] LNetError: 78633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.824555] LNet: Removed LNI 192.168.204.27@tcp2 [ 1799.895788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.971514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.994662] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1801.096853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1801.142531] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1802.085735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 1802.132097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.27@tcp [ 1802.171217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp1 192.168.204.27@tcp1 [ 1802.231253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.27@tcp -d 192.168.204.27@tcp -e network_timeout [ 1802.293988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -e network_timeout [ 1802.341624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -e network_timeout [ 1802.399995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -e network_timeout [ 1802.450492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 1802.505481] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 1802.533122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1802.636444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1802.757520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1803.043029] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 1809.985510] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:29:34 (1790965774) [ 1810.103744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.145314] LNet: 79496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.154091] LNetError: 79496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.163846] LNet: Removed LNI 192.168.204.27@tcp [ 1811.180450] LNet: Removed LNI 192.168.204.27@tcp1 [ 1811.218432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.386869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1811.865660] LNet: 79540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.385854] Key type .llcrypt unregistered [ 1812.390513] Key type ._llcrypt unregistered [ 1825.379601] Key type ._llcrypt registered [ 1825.381813] Key type .llcrypt registered [ 1827.032772] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:29:51 (1790965791) [ 1827.771957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.823432] alg: No test for adler32 (adler32-zlib) [ 1829.069829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.439113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1834.356403] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:29:58 (1790965798) [ 1834.465965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.546674] LNet: 80645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.595226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.761945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.842975] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1836.004928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1836.031721] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 1836.972614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 1837.018570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp1 [ 1837.059769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp1 [ 1844.100403] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:30:08 (1790965808) [ 1845.871477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg427-server [ 1847.934342] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:30:12 (1790965812) [ 1850.487890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg427-server [ 1852.332968] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:30:16 (1790965816) [ 1854.521549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg427-server [ 1856.596530] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:30:21 (1790965821) [ 1858.809313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg427-server [ 1861.239202] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:30:25 (1790965825) [ 1863.202507] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg427-server [ 1865.275980] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:30:29 (1790965829) [ 1867.732459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg427-server [ 1869.741069] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:30:34 (1790965834) [ 1872.105829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg427-server [ 1874.678462] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:30:38 (1790965838) [ 1877.407055] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg427-server [ 1879.761918] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:30:43 (1790965843) [ 1882.000707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg427-server [ 1883.716669] Lustre: DEBUG MARKER: == sanity-lnet test 229: Check NI status with lnetctl net disable/enable ========================================================== 14:30:48 (1790965848) [ 1884.213629] LNet: 84591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.223069] LNetError: 84591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.235077] LNet: Removed LNI 192.168.204.27@tcp [ 1885.292784] LNet: Removed LNI 192.168.204.27@tcp1 [ 1886.240659] Key type .llcrypt unregistered [ 1886.243615] Key type ._llcrypt unregistered [ 1894.024626] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1894.031285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1896.713042] Key type ._llcrypt registered [ 1896.718778] Key type .llcrypt registered [ 1896.972818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1896.995566] alg: No test for adler32 (adler32-zlib) [ 1897.943600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.987391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.013283] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 1898.644914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1898.667347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1901.049954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if test1pg [ 1901.223909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 1901.254498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.27@tcp [ 1901.280772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 10.1.2.3@tcp [ 2081.782273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if test1pg [ 2081.937619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 10.1.2.3@tcp [ 2081.982294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2 [ 2082.164591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 10.1.2.3@tcp [ 2082.199250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 10.1.2.3@tcp [ 2082.223557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2212.831162] INFO: task lnetctl:85432 blocked for more than 120 seconds. [ 2212.834696] Tainted: G O -------- - - 4.18.0rh8.10-debug #2 [ 2212.839610] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2212.844649] task:lnetctl state:D stack:0 pid:85432 ppid:84444 flags:0x80000000 [ 2212.853094] Call Trace: [ 2212.855815] __schedule+0x351/0xcb0 [ 2212.876094] ? wait_for_completion_timeout+0xa9/0x1d0 [ 2212.878846] schedule+0xc0/0x180 [ 2212.879865] schedule_timeout+0xb4/0x190 [ 2212.885691] ? __next_timer_interrupt+0x160/0x160 [ 2212.890728] ? lnet_send+0xc5/0x270 [lnet] [ 2212.894188] ? __prepare_to_swait+0x5b/0x90 [ 2212.896668] wait_for_completion_timeout+0xeb/0x1d0 [ 2212.901042] lnet_ping+0x1dc/0x840 [lnet] [ 2212.904096] ? lnet_peer_ni_show_start+0x9b0/0x9b0 [lnet] [ 2212.908698] ? lnet_genl_send_scalar_list+0x77/0x230 [lnet] [ 2212.912491] lnet_ping_show_dump+0x270/0xe50 [lnet] [ 2212.915721] ? __alloc_skb+0xc2/0x290 [ 2212.917757] ? __build_skb_around+0xab/0xc0 [ 2212.921895] ? __alloc_skb+0x143/0x290 [ 2212.924706] netlink_dump+0x1d4/0x510 [ 2212.926849] __netlink_dump_start+0x20c/0x2f0 [ 2212.929018] genl_family_rcv_msg_dumpit.isra.12+0x9b/0x170 [ 2212.931712] ? genl_rcv_msg+0xd0/0xd0 [ 2212.933691] ? lnet_discover_nid_metadata+0x1260/0x1260 [lnet] [ 2212.936423] ? genl_rcv+0x60/0x60 [ 2212.938516] genl_family_rcv_msg+0x125/0x1f0 [ 2212.940686] ? lnet_net_show_dump+0x1de0/0x1de0 [lnet] [ 2212.943205] ? lnet_mark_ping_buffer_for_update+0x40/0x40 [lnet] [ 2212.945642] ? lnet_discover_nid_metadata+0x1260/0x1260 [lnet] [ 2212.949279] ? lnet_debug_recovery_show_done+0x310/0x310 [lnet] [ 2212.952166] genl_rcv_msg+0x5b/0xd0 [ 2212.953716] ? genl_family_rcv_msg+0x1f0/0x1f0 [ 2212.956318] netlink_rcv_skb+0x62/0x180 [ 2212.958507] genl_rcv+0x34/0x60 [ 2212.960222] netlink_unicast+0x250/0x3e0 [ 2212.962448] netlink_sendmsg+0x3da/0x600 [ 2212.964401] __sock_sendmsg+0x50/0x90 [ 2212.966309] ____sys_sendmsg+0x1ea/0x330 [ 2212.968602] ? copy_msghdr_from_user+0x74/0xb0 [ 2212.970953] ? ____sys_recvmsg+0xcc/0x260 [ 2212.973262] ___sys_sendmsg+0x8c/0xe0 [ 2212.975229] ? copy_msghdr_from_user+0x74/0xb0 [ 2212.977594] ? do_raw_spin_unlock+0x75/0x190 [ 2212.979779] ? idr_get_next_ul+0xbd/0x1a0 [ 2212.982076] ? do_raw_spin_unlock+0x75/0x190 [ 2212.984354] ? __wake_up_common_lock+0x9c/0x130 [ 2212.986626] __sys_sendmsg+0x6b/0xe0 [ 2212.988665] __x64_sys_sendmsg+0x23/0x30 [ 2212.990784] do_syscall_64+0xc1/0x440 [ 2212.992782] entry_SYSCALL_64_after_hwframe+0x49/0xae [ 2212.995248] RIP: 0033:0x7f93b95b8d78 [ 2212.997203] Code: Unable to access opcode bytes at RIP 0x7f93b95b8d4e. [ 2213.000313] RSP: 002b:00007ffccbe78af8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e [ 2213.003785] RAX: ffffffffffffffda RBX: 0000000002011390 RCX: 00007f93b95b8d78 [ 2213.006636] RDX: 0000000000000000 RSI: 00007ffccbe78b30 RDI: 0000000000000003 [ 2213.010251] RBP: 000000000203f4b0 R08: 000000000000f8ff R09: 0000000000000000 [ 2213.014272] R10: 0000000000000000 R11: 0000000000000246 R12: 00000000020112a0 [ 2213.017780] R13: 00007ffccbe78b30 R14: 00007ffccbe78c30 R15: 00007ffccbe78c38 [ 2266.098775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if ens2 [ 2266.234145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2,test1pg [ 2266.346858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2266.360618] LNetError: 85480:0:(lib-move.c:2532:lnet_select_preferred_best_ni()) no path to 192.168.204.27@tcp from net tcp [ 2266.376837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net enable --net tcp --if ens2,test1pg [ 2266.467493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp [ 2266.492762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --if ens2 [ 2266.514730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if ens2 --bogus [ 2266.550795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --if ens2 --net [ 2266.576531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp --if no_such_if [ 2266.599829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net disable --net tcp999 --if ens2 [ 2266.967685] LNet: 85542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.978680] LNetError: 85542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2266.986698] LNet: Removed LNI 10.1.2.3@tcp [ 2268.007951] LNet: Removed LNI 192.168.204.27@tcp [ 2268.521661] Key type .llcrypt unregistered [ 2268.524674] Key type ._llcrypt unregistered [ 2278.985366] Key type ._llcrypt registered [ 2278.987599] Key type .llcrypt registered [ 2280.235888] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:37:24 (1790966244) [ 2280.702871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.721211] alg: No test for adler32 (adler32-zlib) [ 2281.683553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.736355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.757332] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2282.448917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2282.518853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2283.953340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.013567] LNet: 86507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.030668] LNetError: 86507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.053978] LNet: Removed LNI 192.168.204.27@tcp [ 2284.085844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2284.182700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2284.225957] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2285.063486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2285.284510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.326399] LNet: 86652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.335572] LNetError: 86652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.358796] LNet: Removed LNI 192.168.204.27@tcp [ 2285.398195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.475103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.512279] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2286.249835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2286.317812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2286.543294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.579502] LNet: 86800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.587525] LNetError: 86800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.608953] LNet: Removed LNI 192.168.204.27@tcp [ 2286.629849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.696731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2286.731935] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2287.582088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2287.817990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.878259] LNet: Removed LNI 192.168.204.27@tcp [ 2288.911875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.987449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.018407] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2289.822330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2289.898891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2290.091274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.109121] LNet: 87092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.114268] LNet: 87092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2290.122076] LNetError: 87092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.130317] LNetError: 87092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2290.146219] LNet: Removed LNI 192.168.204.27@tcp [ 2290.170210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.241870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2290.264637] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2290.855903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2291.175201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.273601] LNet: Removed LNI 192.168.204.27@tcp [ 2292.302840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.403699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.440149] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2293.382319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2293.514800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2293.823913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.889093] LNet: Removed LNI 192.168.204.27@tcp [ 2294.931936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.006977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2295.029862] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2295.834852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2296.125953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.150306] LNet: 87530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.161690] LNet: 87530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2296.170292] LNetError: 87530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.175036] LNetError: 87530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2297.201461] LNet: Removed LNI 192.168.204.27@tcp [ 2297.250544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.384519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.408569] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2298.381378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2298.514614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2298.890753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2299.948213] LNet: Removed LNI 192.168.204.27@tcp [ 2299.986681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.038661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2300.056300] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2300.791032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2301.174892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.245707] LNet: Removed LNI 192.168.204.27@tcp [ 2302.340343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.579654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2302.644682] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2303.859948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2303.937478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2304.297684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.341503] LNet: 87971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2304.346480] LNet: 87971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2304.353564] LNetError: 87971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2304.359732] LNetError: 87971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2304.371633] LNet: Removed LNI 192.168.204.27@tcp [ 2304.407691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.490731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2304.536482] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2305.404751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2305.864592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.984311] LNet: Removed LNI 192.168.204.27@tcp [ 2307.036391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.196417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.294183] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2308.441398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2308.569268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.669879] LNet: Removed LNI 192.168.204.27@tcp [ 2308.735138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.840853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2308.880793] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2310.590444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2310.825523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.906462] LNet: Removed LNI 192.168.204.27@tcp [ 2310.963271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.084927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 2311.218066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.295245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.410856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 2311.561549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.611807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.716480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 2311.909755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.987478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.094954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 2312.147919] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2313.656970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.692376] LNet: Removed LNI 192.168.204.27@tcp [ 2313.729433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.837809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2322.919272] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:38:07 (1790966287) [ 2323.218387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.281961] LNet: 89146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.290261] LNet: 89146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 2323.314472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.380578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2323.429510] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/180] [ 2325.821717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.875635] LNetError: 89286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.885344] LNetError: 89286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2325.904334] LNet: Removed LNI 192.168.204.27@tcp231 [ 2325.937866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.038953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2326.069324] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/180] [ 2326.076750] LNet: 89294:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2326.930512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2326.973983] LNet: Removed LNI 192.168.204.27@tcp231 [ 2327.005178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2327.077259] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/180] [ 2329.442262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.500748] LNet: Removed LNI 192.168.204.27@tcp231 [ 2329.531928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.620860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2329.650537] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/0] [ 2329.656383] LNet: 89447:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.291470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2330.331130] LNet: Removed LNI 192.168.204.27@tcp231 [ 2330.372780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2330.426657] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/0] [ 2332.286893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.370314] LNet: Removed LNI 192.168.204.27@tcp231 [ 2332.385705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.429983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2332.453261] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/60] [ 2332.461388] LNet: 89598:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2332.812332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2332.840845] LNet: Removed LNI 192.168.204.27@tcp231 [ 2332.873490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2332.904678] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/60] [ 2334.358357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.415140] LNet: Removed LNI 192.168.204.27@tcp231 [ 2334.433676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.550443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2334.584771] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/180] [ 2334.589828] LNet: 89749:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2335.260781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2335.293759] LNet: Removed LNI 192.168.204.27@tcp231 [ 2335.327365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2335.367783] LNet: Added LNI 192.168.204.27@tcp231 [8/256/0/180] [ 2343.483793] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:38:28 (1790966308) [ 2343.738940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.786486] LNet: Removed LNI 192.168.204.27@tcp231 [ 2343.808273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.896805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2343.929163] LNet: Added LNI 192.168.204.27@tcp232 [8/256/0/180] [ 2344.525876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2344.567906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2344.598581] LNet: Removed LNI 192.168.204.27@tcp232 [ 2344.618601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2344.643070] LNet: Added LNI 192.168.204.27@tcp232 [8/256/0/180] [ 2345.261318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2345.292452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2345.324737] LNet: Removed LNI 192.168.204.27@tcp232 [ 2345.348561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2345.394766] LNet: Added LNI 192.168.204.27@tcp232 [8/256/0/180] [ 2346.012484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2346.052435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2346.101444] LNet: Removed LNI 192.168.204.27@tcp232 [ 2346.121700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2346.155782] LNet: Added LNI 192.168.204.27@tcp232 [8/256/0/180] [ 2346.748662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2346.782844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2346.810841] LNet: Removed LNI 192.168.204.27@tcp232 [ 2346.827091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2352.689769] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:38:37 (1790966317) [ 2354.102562] Key type .llcrypt unregistered [ 2354.109726] Key type ._llcrypt unregistered [ 2360.144690] Key type ._llcrypt registered [ 2360.147386] Key type .llcrypt registered [ 2360.401789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2360.420398] alg: No test for adler32 (adler32-zlib) [ 2361.387806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2361.411742] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2370.539589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 2382.311437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.411761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2382.439197] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 2383.834868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 2384.659316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_interrupt -r 2 [ 2384.710087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_interrupt -r 2 [ 2384.752298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_interrupt -r 2 [ 2384.812227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_interrupt -r 2 [ 2384.858819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_interrupt -r 2 [ 2384.909174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_interrupt -r 2 [ 2384.967391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_interrupt -r 2 [ 2385.000764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_interrupt -r 2 [ 2385.042876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2385.063775] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 2385.075251] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 2385.401872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.440503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2385.487950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.520767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.549713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2385.576707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.602463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.633841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.661635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.704290] LNet: 92273:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.743230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.813366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2385.867929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2385.903443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2385.967541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.017026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.517939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_dropped -r 2 [ 2386.562702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_dropped -r 2 [ 2386.638682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_dropped -r 2 [ 2386.715803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_dropped -r 2 [ 2386.773792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_dropped -r 2 [ 2386.816169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_dropped -r 2 [ 2386.866741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_dropped -r 2 [ 2386.922937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_dropped -r 2 [ 2387.006877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2387.042953] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 2387.413216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.442791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2387.469441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.501262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.525997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2387.560428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.612875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.654879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.695550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.726909] LNet: 92396:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.786426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.850113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.907567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.947458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.984740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2388.011325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2388.717686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_aborted -r 2 [ 2388.750407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_aborted -r 2 [ 2388.782598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_aborted -r 2 [ 2388.813642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_aborted -r 2 [ 2388.845917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_aborted -r 2 [ 2388.883842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_aborted -r 2 [ 2388.947440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_aborted -r 2 [ 2388.981858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_aborted -r 2 [ 2389.024839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2389.051558] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 2389.265808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2389.294372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2389.324276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2389.355679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2389.380585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2389.407584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2389.444811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2389.500665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2389.549988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2389.573705] LNet: 92518:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2389.593863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2389.622905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2389.655739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2389.693703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2389.733454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2389.769240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2390.231272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_no_route -r 2 [ 2390.269463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_no_route -r 2 [ 2390.301364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_no_route -r 2 [ 2390.339936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_no_route -r 2 [ 2390.376530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_no_route -r 2 [ 2390.418256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_no_route -r 2 [ 2390.458122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_no_route -r 2 [ 2390.497889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_no_route -r 2 [ 2390.536958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2390.849869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2390.891872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2390.940410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2390.975872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2391.007463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2391.045234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2391.083788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2391.123108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2391.153374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2391.184769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2391.219184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2391.267674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2391.321221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2391.356563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2391.393736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2391.988630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e local_timeout -r 2 [ 2392.040062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e local_timeout -r 2 [ 2392.083425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e local_timeout -r 2 [ 2392.133713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e local_timeout -r 2 [ 2392.174238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e local_timeout -r 2 [ 2392.223025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e local_timeout -r 2 [ 2392.272353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e local_timeout -r 2 [ 2392.321261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e local_timeout -r 2 [ 2392.387451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2392.432162] LNet: There was an unexpected network error while writing to 192.168.204.127: rc = -22 [ 2392.450569] LNet: Skipped 1 previous similar message [ 2392.677503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2392.714653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2392.746423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2392.785877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2392.818476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2392.861637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2392.892877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2392.925652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2392.958104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2392.992758] LNet: 92763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2393.006400] LNet: 92763:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2393.025611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2393.059131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2393.084887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2393.111484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2393.135916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2393.173469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2393.685317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp -m GET -e remote_dropped -r 2 [ 2393.712776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.127@tcp1 -m GET -e remote_dropped -r 2 [ 2393.740702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp -m GET -e remote_dropped -r 2 [ 2393.771372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp -d 192.168.204.27@tcp1 -m GET -e remote_dropped -r 2 [ 2393.804626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp -m GET -e remote_dropped -r 2 [ 2393.843166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.27@tcp1 -d 192.168.204.127@tcp1 -m GET -e remote_dropped -r 2 [ 2393.900970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp -m GET -e remote_dropped -r 2 [ 2393.969140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.27@tcp1 -d 192.168.204.27@tcp1 -m GET -e remote_dropped -r 2 [ 2394.058131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.27@tcp 192.168.204.127@tcp [ 2394.095269] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.127@tcp [ 2394.371858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2394.426658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2394.467481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2394.509482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2394.548570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2394.575774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2394.606808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2394.640547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2394.668853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2394.700970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2394.740835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2394.780342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2394.843900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2394.887552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2394.931745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2400.477293] LNet: 92959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.489228] LNetError: 92959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.504165] LNet: Removed LNI 192.168.204.27@tcp [ 2400.537420] LNet: Removed LNI 192.168.204.27@tcp1 [ 2401.018909] Key type .llcrypt unregistered [ 2401.023517] Key type ._llcrypt unregistered [ 2416.958868] Key type ._llcrypt registered [ 2416.963550] Key type .llcrypt registered [ 2418.721273] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:39:42 (1790966382) [ 2419.277969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2419.304225] alg: No test for adler32 (adler32-zlib) [ 2420.328803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.450912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.491159] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2421.619891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2421.655576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.27@tcp [ 2421.679363] LNetError: 93942:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.27@tcp to recovery [ 2421.825965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.27@tcp [ 2421.983799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2429.463279] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:39:53 (1790966393) [ 2429.583498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.602740] LNet: 94535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.609967] LNetError: 94535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.641058] LNet: Removed LNI 192.168.204.27@tcp [ 2429.679951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.735641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.762478] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2430.595967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2430.621688] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 2431.670874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.27@tcp [ 2431.695845] LNetError: 94807:0:(api-ni.c:4484:lnet_ni_set_healthv()) manually adding local NI 192.168.204.27@tcp to recovery [ 2431.776500] LNet: 1 local NIs in recovery (showing 1): 192.168.204.27@tcp [ 2431.794233] LNet: There was an unexpected network error while writing to 192.168.204.27: rc = -22 [ 2432.973733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.27@tcp [ 2441.053861] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:40:05 (1790966405) [ 2443.105326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg427-server [ 2445.555838] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:40:09 (1790966409) [ 2448.351637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg427-server [ 2450.665904] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:40:14 (1790966414) [ 2452.888737] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg427-server [ 2454.564942] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:40:19 (1790966419) [ 2454.948333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.978738] LNet: 96456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.991190] LNetError: 96456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2455.005405] LNet: Removed LNI 192.168.204.27@tcp [ 2455.018769] LNet: Removed LNI 192.168.204.27@tcp1 [ 2455.086903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.196891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2455.235138] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2455.937483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2456.007888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2456.046944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2456.147371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2456.196958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2456.273344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2464.544526] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:40:28 (1790966428) [ 2464.775795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2464.794922] LNet: 97195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.799955] LNetError: 97195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.805043] LNetError: 97195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2464.824373] LNet: Removed LNI 192.168.204.27@tcp [ 2464.859081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.937405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2464.959315] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2474.108797] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:40:38 (1790966438) [ 2474.699068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2474.772661] LNet: 97907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.797112] LNetError: 97907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2474.839216] LNet: Removed LNI 192.168.204.27@tcp [ 2474.888928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.976775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2475.031274] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2486.004635] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:40:50 (1790966450) [ 2488.142936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2490.744380] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:40:54 (1790966454) [ 2491.656715] LNet: 98956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.667855] LNetError: 98956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2491.691673] LNet: Removed LNI 192.168.204.27@tcp [ 2492.181126] Key type .llcrypt unregistered [ 2492.188188] Key type ._llcrypt unregistered [ 2499.962866] Key type ._llcrypt registered [ 2499.968513] Key type .llcrypt registered [ 2500.529739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2500.568338] alg: No test for adler32 (adler32-zlib) [ 2501.525716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2501.550445] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2511.157066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 2517.799895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.127@tcp [ 2518.263055] LNet: 99369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2518.274899] LNetError: 99369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.315299] LNet: Removed LNI 192.168.204.27@tcp [ 2518.850760] Key type .llcrypt unregistered [ 2518.853048] Key type ._llcrypt unregistered [ 2533.944937] Key type ._llcrypt registered [ 2533.950271] Key type .llcrypt registered [ 2535.723880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2538.393526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2540.771815] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:41:44 (1790966504) [ 2542.797876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2545.799931] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:41:49 (1790966509) [ 2548.067354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg427-server [ 2549.920324] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:41:54 (1790966514) [ 2552.056558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg427-server [ 2554.576358] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:41:58 (1790966518) [ 2556.726139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2556.752445] alg: No test for adler32 (adler32-zlib) [ 2557.869445] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 2558.150556] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2559.848960] Key type lgssc registered [ 2561.624277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2584.199385] Key type lgssc unregistered [ 2584.571769] LNet: 102611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2584.586760] LNetError: 102611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2584.621163] LNet: Removed LNI 192.168.204.27@tcp [ 2585.671261] Key type .llcrypt unregistered [ 2585.674504] Key type ._llcrypt unregistered [ 2602.335223] Key type ._llcrypt registered [ 2602.337103] Key type .llcrypt registered [ 2603.654917] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:42:48 (1790966568) [ 2604.138444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2604.174503] alg: No test for adler32 (adler32-zlib) [ 2605.128185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2605.202370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2605.242423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2605.295513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2605.357858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2611.801284] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:42:56 (1790966576) [ 2611.976446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2612.009787] LNet: 104043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2612.039516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2612.107139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2612.128828] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2612.995555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2613.016522] LNet: 104189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2613.022476] LNetError: 104189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2613.036982] LNet: Removed LNI 192.168.204.27@tcp [ 2613.056415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2613.108875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2613.133389] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2613.653704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2613.673944] LNetError: 104217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2613.684429] LNet: Removed LNI 192.168.204.27@tcp [ 2613.711099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2613.784884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2613.806533] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2621.237731] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:43:05 (1790966585) [ 2621.323577] LNet: 104807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2621.328517] LNet: 104807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2621.342349] LNetError: 104807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2621.365804] LNet: Removed LNI 192.168.204.27@tcp [ 2622.318538] Key type .llcrypt unregistered [ 2622.323194] Key type ._llcrypt unregistered [ 2630.243706] Key type ._llcrypt registered [ 2630.252285] Key type .llcrypt registered [ 2630.486362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2630.513349] alg: No test for adler32 (adler32-zlib) [ 2631.447686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2631.550559] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2631.558128] LNet: *** cfs_fail_loc=e003, val=100*** [ 2632.303378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2632.321956] LNet: *** cfs_fail_loc=e003, val=100*** [ 2632.324284] LNet: Skipped 2 previous similar messages [ 2632.327990] Ping reply received with mapping: [ 2632.330139] found metadata reply NI[192.168.204.27@tcp] [ 2632.332543] 1b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2632.335206] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2632.337638] 5a 5a 5a 5a 5a 5a 5a 5a [ 2632.541553] LNet: 105208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.547428] LNetError: 105208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.564230] LNet: Removed LNI 192.168.204.27@tcp [ 2633.242442] Key type .llcrypt unregistered [ 2633.250201] Key type ._llcrypt unregistered [ 2643.718649] Key type ._llcrypt registered [ 2643.721146] Key type .llcrypt registered [ 2644.934922] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:43:29 (1790966609) [ 2645.829627] Key type .llcrypt unregistered [ 2645.832096] Key type ._llcrypt unregistered [ 2652.795280] Key type ._llcrypt registered [ 2652.797818] Key type .llcrypt registered [ 2653.059814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2653.086518] alg: No test for adler32 (adler32-zlib) [ 2654.191235] LNetError: 106248:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2654.751408] Key type .llcrypt unregistered [ 2654.757462] Key type ._llcrypt unregistered [ 2660.690832] Key type ._llcrypt registered [ 2660.692978] Key type .llcrypt registered [ 2662.181519] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:43:46 (1790966626) [ 2662.892288] Key type .llcrypt unregistered [ 2662.896356] Key type ._llcrypt unregistered [ 2670.627094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2670.958230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2670.960806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2670.964322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2673.613160] Key type ._llcrypt registered [ 2673.621731] Key type .llcrypt registered [ 2673.939300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2673.977950] alg: No test for adler32 (adler32-zlib) [ 2675.013663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2675.119553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2686.208713] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:44:11 (1790966651) [ 2686.448985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2686.513449] LNet: 108056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.533655] LNetError: 108056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2686.580347] LNet: Removed LNI 10.1.2.3@tcp [ 2686.626260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2687.372786] Key type .llcrypt unregistered [ 2687.375942] Key type ._llcrypt unregistered [ 2694.235450] Key type ._llcrypt registered [ 2694.238522] Key type .llcrypt registered [ 2694.506321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2694.533476] alg: No test for adler32 (adler32-zlib) [ 2695.509533] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2695.619456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2695.854584] LNet: 108345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2695.863321] LNetError: 108345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2696.935891] LNet: Removed LNI 192.168.204.27@tcp [ 2697.889961] Key type .llcrypt unregistered [ 2697.893793] Key type ._llcrypt unregistered [ 2712.143079] Key type ._llcrypt registered [ 2712.148894] Key type .llcrypt registered [ 2714.659272] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:44:37 (1790966677) [ 2715.872502] Key type .llcrypt unregistered [ 2715.877119] Key type ._llcrypt unregistered [ 2723.893078] Key type ._llcrypt registered [ 2723.902790] Key type .llcrypt registered [ 2724.356885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2724.405804] alg: No test for adler32 (adler32-zlib) [ 2736.708630] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:45:00 (1790966700) [ 2736.927521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2737.047472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2737.164370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2737.213403] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2738.508772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2738.582669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2738.635852] LNet: Removed LNI 192.168.204.27@tcp [ 2738.962749] LNet: 110112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2739.362398] Key type .llcrypt unregistered [ 2739.364178] Key type ._llcrypt unregistered [ 2751.046406] Key type ._llcrypt registered [ 2751.048702] Key type .llcrypt registered [ 2752.286093] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:45:16 (1790966716) [ 2752.672508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.691486] alg: No test for adler32 (adler32-zlib) [ 2753.576116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2753.624891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2753.642021] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2754.316866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2754.346047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.27@tcp [ 2754.373936] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2754.382134] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2754.386920] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2754, 0, 0 [ 2754.393777] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2754.404937] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2754.413507] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2754.420156] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2754, 0, 0 [ 2754.427146] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2754.428392] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2754.435624] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2754.446282] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2754, 0, 2804 [ 2754.454493] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2754.460610] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2754.467648] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2754.473154] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2754, 0, 2804 [ 2754.478807] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2761.113386] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:45:25 (1790966725) [ 2761.796803] LNet: 111669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.816568] LNetError: 111669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.853314] LNet: Removed LNI 192.168.204.27@tcp [ 2762.754305] Key type .llcrypt unregistered [ 2762.756496] Key type ._llcrypt unregistered [ 2771.003997] Key type ._llcrypt registered [ 2771.006338] Key type .llcrypt registered [ 2771.167497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2771.185385] alg: No test for adler32 (adler32-zlib) [ 2772.255654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2772.319495] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2780.650896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 2793.117891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2793.168989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2793.215203] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 2794.517928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 2794.712841] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2794.898772] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2794.905618] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2794.908487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2797.530534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2797.568322] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2809.255416] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:46:13 (1790966773) [ 2809.412818] LNet: 113184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2809.430948] LNetError: 113184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2809.448058] LNet: Removed LNI 192.168.204.27@tcp [ 2809.454880] LNet: Removed LNI 192.168.204.27@tcp1 [ 2809.487577] LNet: Removed LNI 10.1.2.3@tcp99 [ 2810.151767] Key type .llcrypt unregistered [ 2810.154868] Key type ._llcrypt unregistered [ 2817.922202] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2817.934840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2820.679317] Key type ._llcrypt registered [ 2820.687867] Key type .llcrypt registered [ 2821.083567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2821.128736] alg: No test for adler32 (adler32-zlib) [ 2822.219957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2822.282519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2822.336720] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2823.344244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2823.370336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2826.615837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2826.680175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2826.754593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2826.771142] LNetError: 113937:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2826.800314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2826.843888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2827.375116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2827.448296] LNet: 113957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.478282] LNetError: 113957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2827.491902] LNet: Removed LNI 10.1.2.3@tcp [ 2828.519415] LNet: Removed LNI 192.168.204.27@tcp [ 2829.460605] Key type .llcrypt unregistered [ 2829.465695] Key type ._llcrypt unregistered [ 2845.445329] Key type ._llcrypt registered [ 2845.447205] Key type .llcrypt registered [ 2847.650184] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:46:51 (1790966811) [ 2848.994355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2849.038148] alg: No test for adler32 (adler32-zlib) [ 2850.232651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2850.346936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2850.438472] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2851.729149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 2851.864978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg427-client@tcp [ 2859.900786] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:47:03 (1790966823) [ 2860.109448] LNet: 115502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.129955] LNetError: 115502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.206487] LNet: Removed LNI 192.168.204.27@tcp [ 2861.243335] Key type .llcrypt unregistered [ 2861.251067] Key type ._llcrypt unregistered [ 2869.834391] Key type ._llcrypt registered [ 2869.836914] Key type .llcrypt registered [ 2870.004994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2870.021571] alg: No test for adler32 (adler32-zlib) [ 2871.006591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2871.048257] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2871.104458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2879.811462] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:47:24 (1790966844) [ 2882.333989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2884.901996] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:47:29 (1790966849) [ 2886.723843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2890.147590] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:47:33 (1790966853) [ 2890.484966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2890.548553] LNet: 116943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.570898] LNetError: 116943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.612476] LNet: Removed LNI 192.168.204.27@tcp [ 2890.666426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2890.835816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2890.920597] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2892.023902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2892.111324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2902.116997] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:47:46 (1790966866) [ 2902.339306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2902.363607] LNet: 117667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.381214] LNetError: 117667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2902.401866] LNet: Removed LNI 192.168.204.27@tcp [ 2902.425591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2902.456584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2902.479171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2903.376417] Key type .llcrypt unregistered [ 2903.379564] Key type ._llcrypt unregistered [ 2917.161908] Key type ._llcrypt registered [ 2917.175172] Key type .llcrypt registered [ 2918.735056] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:48:03 (1790966883) [ 2919.307660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2919.338402] alg: No test for adler32 (adler32-zlib) [ 2920.389446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2920.479916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2920.519186] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2921.465818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2921.502987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2928.611468] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:48:13 (1790966893) [ 2928.790910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2928.826477] LNet: 119236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2928.835888] LNetError: 119236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2928.893260] LNet: Removed LNI 192.168.204.27@tcp [ 2928.986929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2929.068778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2929.097589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2936.022470] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:48:20 (1790966900) [ 2936.122033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2936.189304] LNet: 119824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2936.255544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2936.386952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2936.651943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2936.670785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2936.705839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2936.927756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2936.949983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2936.998985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2937.025392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2937.067636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2937.352947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2937.391795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2937.476878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2938.078357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2938.133181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2945.590187] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:48:29 (1790966909) [ 2945.879614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2945.922791] LNet: 120469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2945.930352] LNet: 120469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2945.990896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2946.051882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2946.075547] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2947.027948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2947.096160] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2948.564628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2948.649216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2949.054403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2949.111819] LNet: 120752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2949.118753] LNetError: 120752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2950.175377] LNet: Removed LNI 192.168.204.27@tcp [ 2950.184251] LNetError: 120752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2950.212072] LNet: Removed LNI 192.168.204.27@tcp2 [ 2950.321985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2950.425591] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2950.452377] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2950.467225] LNet: 120759:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2951.596901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2952.019405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2953.057321] LNet: Removed LNI 192.168.204.27@tcp [ 2953.062264] LNetError: 120796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2953.072665] LNetError: 120796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2953.096088] LNet: Removed LNI 192.168.204.27@tcp2 [ 2953.270767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2953.398803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2953.470364] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2954.773656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2954.818285] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2956.441987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2956.534554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2956.658121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2957.094837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2957.136697] LNet: 121081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2957.150526] LNet: 121081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2957.162421] LNetError: 121081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2957.176848] LNet: Removed LNI 192.168.204.27@tcp [ 2958.251214] LNet: Removed LNI 192.168.204.27@tcp2 [ 2958.357900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2958.555113] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2958.584778] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2958.616302] LNet: 121087:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2959.780457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2960.153170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2960.174887] LNet: Removed LNI 192.168.204.27@tcp [ 2961.192559] LNet: Removed LNI 192.168.204.27@tcp2 [ 2961.280036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2961.361095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2961.401935] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2962.916238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2962.942461] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2964.686623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2964.778458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2965.084317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2966.112467] LNet: Removed LNI 192.168.204.27@tcp [ 2966.117796] LNetError: 121404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2966.145854] LNetError: 121404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2966.181262] LNet: Removed LNI 192.168.204.27@tcp2 [ 2966.221373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2966.350925] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2966.367956] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2966.378641] LNet: 121411:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2967.454859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2967.818636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2967.844252] LNet: 121448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.846813] LNet: 121448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2968.863438] LNet: Removed LNI 192.168.204.27@tcp [ 2968.878812] LNet: Removed LNI 192.168.204.27@tcp2 [ 2968.929126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2969.056577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2969.119291] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2970.440620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2970.490525] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2971.811684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2971.844201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2971.896436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2972.082461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2973.151521] LNet: Removed LNI 192.168.204.27@tcp [ 2974.187390] LNet: Removed LNI 192.168.204.27@tcp2 [ 2974.206822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2974.270188] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2974.275658] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2974.281321] LNet: 121739:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2974.731743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2974.968478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2975.001254] LNet: Removed LNI 192.168.204.27@tcp [ 2976.043707] LNet: Removed LNI 192.168.204.27@tcp2 [ 2976.074585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2976.140762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2976.159806] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2977.012749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2977.061710] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2978.835517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2978.899909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2979.023605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2979.648828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2980.703388] LNet: Removed LNI 192.168.204.27@tcp [ 2980.717639] LNet: Removed LNI 192.168.204.27@tcp2 [ 2980.778408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2980.848585] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2980.867695] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2980.879554] LNet: 122079:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2981.827423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2982.231352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2982.259970] LNetError: 122132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2982.269883] LNetError: 122132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 2982.284046] LNet: Removed LNI 192.168.204.27@tcp [ 2982.298271] LNet: Removed LNI 192.168.204.27@tcp2 [ 2982.328964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2982.445333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2982.503186] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2983.490905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2983.531409] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2984.655834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2984.728575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2984.805895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2984.911352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2985.310245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2985.326386] LNet: 122433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2985.331400] LNet: 122433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2986.399788] LNet: Removed LNI 192.168.204.27@tcp [ 2987.433053] LNet: Removed LNI 192.168.204.27@tcp2 [ 2987.475245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2987.557631] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 2987.565712] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 2988.204898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 2996.667039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2998.488735] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:49:23 (1790966963) [ 2998.557310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2998.573740] LNet: Removed LNI 192.168.204.27@tcp [ 2999.590217] LNet: Removed LNI 192.168.204.27@tcp2 [ 2999.639657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3000.075596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3000.108413] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3000.994294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3001.016803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3001.992087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3002.041325] LNet: Removed LNI 10.1.2.3@tcp [ 3002.053739] LNet: Removed LNI 192.168.204.27@tcp [ 3002.101910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3002.179817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3002.241065] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3002.249678] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3003.078992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3003.110537] LNet: Removed LNI 192.168.204.27@tcp [ 3003.118800] LNet: Removed LNI 10.1.2.3@tcp [ 3011.428631] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 14:49:35 (1790966975) [ 3011.970247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3012.063045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3012.194273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3012.260252] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3013.316438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3013.342868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3013.371399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.403770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.431486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.456236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.481941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.507378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.528702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.561779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.590332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.625847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3013.744458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3023.137402] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 14:49:46 (1790966986) [ 3023.305651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3023.361814] LNet: 124836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3023.377656] LNet: 124836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3023.405849] LNetError: 124836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3023.417949] LNetError: 124836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3023.459585] LNet: Removed LNI 192.168.204.27@tcp [ 3023.541737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3023.629909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3023.654484] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3024.699990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 3024.777760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 3024.875429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3024.938507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3024.981676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.077773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.131842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.174881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.222295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.273529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.312964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.351840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.462507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3025.539985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.568506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.601144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.649525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.708881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.758353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.812670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.881493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3025.928373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3026.013631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.27@tcp [ 3035.121635] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:49:58 (1790966998) [ 3035.348641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3036.456027] LNet: Removed LNI 192.168.204.27@tcp [ 3036.517981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3036.701828] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3036.961813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3036.969654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3036.979892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3039.355363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3039.406529] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3040.501283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3040.523645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3043.501875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 3043.582288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3043.669748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.27@tcp -m PUT -e local_timeout -r 1 [ 3043.763859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3043.831090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3043.837737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3044.271986] LNetError: 125672:0:(lib-move.c:4823:lnet_parse()) 192.168.204.27@tcp, src 192.168.204.27@tcp: Dropping PUT (error -108 looking up sender) [ 3045.279992] LNet: Removed LNI 10.1.2.3@tcp [ 3046.323414] LNet: Removed LNI 192.168.204.27@tcp [ 3046.903260] Key type .llcrypt unregistered [ 3046.909780] Key type ._llcrypt unregistered [ 3062.690651] Key type ._llcrypt registered [ 3062.695056] Key type .llcrypt registered [ 3064.339654] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:50:28 (1790967028) [ 3064.923755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3064.961479] alg: No test for adler32 (adler32-zlib) [ 3065.943405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3066.384723] LNet: 126995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3067.334389] Key type .llcrypt unregistered [ 3067.336751] Key type ._llcrypt unregistered [ 3075.076954] Key type ._llcrypt registered [ 3075.078970] Key type .llcrypt registered [ 3075.504747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3075.530856] alg: No test for adler32 (adler32-zlib) [ 3076.606055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3076.701362] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3084.492970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 3084.818798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3085.470738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.127@tcp [ 3085.750406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3085.782425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3085.808288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3085.869365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3085.939684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3085.992124] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.127@tcp [ 3085.998768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3086.051961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3086.085358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3086.111783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3086.136822] LNet: 127448:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3086.154551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3086.180143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3086.203849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3086.231511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3086.262805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3086.299869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3087.265478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3087.708448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3087.764288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3087.835960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3087.901261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3087.958695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3088.008403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3088.079277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3088.133377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3088.193834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3088.225610] LNet: 127567:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3088.268357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3088.355706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3088.409976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3088.445925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3088.504339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3088.588539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3089.667162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.127@tcp [ 3090.164941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3090.245418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3090.335447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3090.439828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3090.490879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3090.582297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3090.642349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3090.706241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3090.783765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3090.827773] LNet: 127687:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3090.865764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3090.970899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3091.053732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3091.143275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3091.275868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3091.298514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3091.860609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3092.103586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3092.163397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3092.207836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3092.253425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3092.303598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3092.357441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3092.398676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3092.429623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3092.462560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3092.501684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3092.550532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3092.591490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3092.633540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3092.678271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3092.738529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3093.095493] LNet: 127852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3093.113056] LNetError: 127852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3093.134192] LNet: Removed LNI 192.168.204.27@tcp [ 3093.570901] Key type .llcrypt unregistered [ 3093.574159] Key type ._llcrypt unregistered [ 3119.585290] Key type ._llcrypt registered [ 3119.592637] Key type .llcrypt registered [ 3121.395903] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:51:25 (1790967085) [ 3121.953582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3121.965351] alg: No test for adler32 (adler32-zlib) [ 3122.903878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3123.471764] LNet: 128754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3124.193371] Key type .llcrypt unregistered [ 3124.198579] Key type ._llcrypt unregistered [ 3131.278966] Key type ._llcrypt registered [ 3131.281747] Key type .llcrypt registered [ 3131.619910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3131.663264] alg: No test for adler32 (adler32-zlib) [ 3132.543405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3132.562380] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3140.732125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 3150.723200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3150.768156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3150.785074] LNet: Added LNI 192.168.204.27@tcp1 [8/256/0/180] [ 3151.604372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.127@tcp [ 3151.830185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3152.678182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.127@tcp [ 3153.010433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3153.057020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3153.060698] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.127@tcp [ 3153.124403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3153.178946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3153.237926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3153.287966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3153.342567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3153.418634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3153.453880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3153.488164] LNet: 129415:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3153.510458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3153.557450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3153.612474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3153.707889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3153.769586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3153.840992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3154.610723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3155.031954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3155.082243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3155.147855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3155.183192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3155.235907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3155.297674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3155.343158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3155.379967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3155.419424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3155.440759] LNet: 129535:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3155.465449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3155.491740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3155.571083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3155.635876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3155.707142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3155.811055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3156.893521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.127@tcp [ 3157.308417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3157.382889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3157.447801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3157.536376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3157.581377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3157.668699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3157.716052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3157.768867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3157.811285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3157.833215] LNet: 129654:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3157.856906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3157.914653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3157.989270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3158.068034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3158.113931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3158.203401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3159.417511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3159.882639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3160.031114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 3160.167252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3160.263884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3160.373633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 3160.558800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3160.628384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3160.742729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3160.835598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3160.874341] LNet: 129774:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3160.903463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3161.008163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3161.121092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3161.241533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3161.332236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3161.383605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3168.844411] LNet: 129855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3168.858567] LNetError: 129855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3168.870469] LNet: Removed LNI 192.168.204.27@tcp [ 3168.886956] LNet: Removed LNI 192.168.204.27@tcp1 [ 3169.621493] Key type .llcrypt unregistered [ 3169.630234] Key type ._llcrypt unregistered [ 3190.418650] Key type ._llcrypt registered [ 3190.421576] Key type .llcrypt registered [ 3192.332245] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 14:52:36 (1790967156) [ 3192.918422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3192.951256] alg: No test for adler32 (adler32-zlib) [ 3193.934798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3194.038790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3194.096962] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3195.179379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3195.208817] LNet: Added LNI 192.168.204.27@tcp2 [8/256/0/180] [ 3196.332301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.27@tcp [ 3196.437389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 3197.471559] LNet: Removed LNI 192.168.204.27@tcp [ 3205.211163] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:52:49 (1790967169) [ 3206.768518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3208.885994] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:52:53 (1790967173) [ 3216.607690] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:53:00 (1790967180) [ 3226.752364] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:53:10 (1790967190) [ 3235.771635] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:53:20 (1790967200) [ 3244.647469] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:53:28 (1790967208) [ 3256.680859] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:53:40 (1790967220) [ 3269.309643] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:53:53 (1790967233) [ 3279.426140] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:54:03 (1790967243) [ 3289.285092] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:54:13 (1790967253) [ 3298.605553] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:54:22 (1790967262) [ 3308.633815] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:54:32 (1790967272) [ 3320.175775] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:54:44 (1790967284) [ 3331.681903] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:54:56 (1790967296) [ 3341.240525] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:55:05 (1790967305) [ 3349.065437] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:55:13 (1790967313) [ 3356.558979] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:55:21 (1790967321) [ 3364.904156] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:55:29 (1790967329) [ 3373.582512] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:55:38 (1790967338) [ 3373.781378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3373.818192] LNet: 141712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3373.828090] LNetError: 141712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3373.864907] LNet: Removed LNI 192.168.204.27@tcp2 [ 3373.968219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3374.202574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3374.229824] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3375.527837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3375.556315] LNet: 141856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3375.568516] LNetError: 141856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3375.596740] LNet: Removed LNI 192.168.204.27@tcp [ 3375.632758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3375.743272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3375.789250] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3383.489164] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:55:47 (1790967347) [ 3392.739335] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:55:57 (1790967357) [ 3402.265499] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:56:06 (1790967366) [ 3412.834800] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:56:17 (1790967377) [ 3421.289056] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:56:26 (1790967386) [ 3433.044915] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:56:36 (1790967396) [ 3445.817854] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:56:48 (1790967408) [ 3455.848104] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:56:59 (1790967419) [ 3466.091241] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:57:10 (1790967430) [ 3473.783721] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:57:18 (1790967438) [ 3481.122678] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:57:25 (1790967445) [ 3489.359952] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:57:33 (1790967453) [ 3498.470330] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3180 sec ========== 14:57:42 (1790967462) [ 3498.859663] LNet: 149434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3498.865952] LNetError: 149434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3498.882203] LNet: Removed LNI 192.168.204.27@tcp [ 3499.635517] Key type .llcrypt unregistered [ 3499.647466] Key type ._llcrypt unregistered [ 3513.568110] Key type ._llcrypt registered [ 3513.577889] Key type .llcrypt registered [ 3514.426192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3514.459051] alg: No test for adler32 (adler32-zlib) [ 3515.619206] Lustre: Lustre: Build Version: 2.17.59_2_g3d9f037 [ 3515.854157] LNet: Added LNI 192.168.204.27@tcp [8/256/0/180] [ 3517.519417] Key type lgssc registered [ 3519.050542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3623.090298] Lustre: Mounted lustre-client - version 2.17.59_2_g3d9f037 [ 3629.866995] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3638.654594] Lustre: Unmounted lustre-client [ 3702.014553] Key type lgssc unregistered [ 3702.337846] LNet: 152966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3702.342254] LNetError: 152966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3702.361379] LNet: Removed LNI 192.168.204.27@tcp [ 3703.175169] Key type .llcrypt unregistered [ 3703.180774] Key type ._llcrypt unregistered