[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 509318937 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003088] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008313] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.011193] LSM: Security Framework initializing [ 0.012063] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014077] *** VALIDATE selinux *** [ 0.023471] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028740] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029183] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031143] *** VALIDATE tmpfs *** [ 0.033345] *** VALIDATE proc *** [ 0.034280] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036303] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038120] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039014] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040040] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.045397] debug: unmapping init [mem 0xffffffff94c59000-0xffffffff94c60fff] [ 0.047340] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049657] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050034] ... version: 2 [ 0.051021] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053020] ... value mask: 0000ffffffffffff [ 0.054016] ... max period: 00007fffffffffff [ 0.055024] ... fixed-purpose events: 3 [ 0.056012] ... event mask: 000000070000000f [ 0.057440] rcu: Hierarchical SRCU implementation. [ 0.059887] smp: Bringing up secondary CPUs ... [ 0.060629] x86: Booting SMP configuration: [ 0.061038] .... node #0, CPUs: #1 #2 #3 [ 0.064441] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149138] node 0 deferred pages initialised in 76ms [ 0.152374] devtmpfs: initialized [ 0.154235] x86/mm: Memory block size: 128MB [ 0.156911] gcov: version magic: 0x41383552 [ 0.157705] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162418] pinctrl core: initialized pinctrl subsystem [ 0.164159] [ 0.164714] ************************************************************* [ 0.167020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ** ** [ 0.172012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174011] ** ** [ 0.176017] ** This means that this kernel is built to expose internal ** [ 0.179014] ** IOMMU data structures, which may compromise security on ** [ 0.181013] ** your system. ** [ 0.183013] ** ** [ 0.186015] ** If you see this message and you are not debugging the ** [ 0.188013] ** kernel, report this immediately to your vendor! ** [ 0.191015] ** ** [ 0.193013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.195076] ************************************************************* [ 0.198760] NET: Registered protocol family 16 [ 0.200492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210034] cpuidle: using governor menu [ 0.212227] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214525] PCI: Using configuration type 1 for base access [ 0.215136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.230196] cryptd: max_cpu_qlen set to 1000 [ 0.234284] ACPI: Added _OSI(Module Device) [ 0.236015] ACPI: Added _OSI(Processor Device) [ 0.238016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.239013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.244720] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.250598] ACPI: Interpreter enabled [ 0.251071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253015] ACPI: Using IOAPIC for interrupt routing [ 0.254116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257511] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269265] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.271042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281447] acpiphp: Slot [2] registered [ 0.283096] acpiphp: Slot [5] registered [ 0.284072] acpiphp: Slot [6] registered [ 0.285047] acpiphp: Slot [3] registered [ 0.286080] acpiphp: Slot [4] registered [ 0.287096] acpiphp: Slot [7] registered [ 0.288098] acpiphp: Slot [8] registered [ 0.289099] acpiphp: Slot [9] registered [ 0.291106] acpiphp: Slot [10] registered [ 0.292193] acpiphp: Slot [11] registered [ 0.294114] acpiphp: Slot [12] registered [ 0.295174] acpiphp: Slot [13] registered [ 0.297133] acpiphp: Slot [14] registered [ 0.298113] acpiphp: Slot [15] registered [ 0.300115] acpiphp: Slot [16] registered [ 0.301100] acpiphp: Slot [17] registered [ 0.303121] acpiphp: Slot [18] registered [ 0.304090] acpiphp: Slot [19] registered [ 0.306117] acpiphp: Slot [20] registered [ 0.308144] acpiphp: Slot [21] registered [ 0.309082] acpiphp: Slot [22] registered [ 0.310083] acpiphp: Slot [23] registered [ 0.312156] acpiphp: Slot [24] registered [ 0.314139] acpiphp: Slot [25] registered [ 0.315237] acpiphp: Slot [26] registered [ 0.317123] acpiphp: Slot [27] registered [ 0.318104] acpiphp: Slot [28] registered [ 0.320094] acpiphp: Slot [29] registered [ 0.321104] acpiphp: Slot [30] registered [ 0.322130] acpiphp: Slot [31] registered [ 0.323079] PCI host bridge to bus 0000:00 [ 0.325021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.329020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.337819] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.341332] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.348018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.352059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.355029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.359029] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.361593] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364928] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368180] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370975] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.387894] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.403166] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.434024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.452085] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.462018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.480016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.490025] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.493428] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.495362] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.498472] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.500219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.506108] iommu: Default domain type: Passthrough [ 0.508467] SCSI subsystem initialized [ 0.511218] ACPI: bus type USB registered [ 0.513129] usbcore: registered new interface driver usbfs [ 0.516115] usbcore: registered new interface driver hub [ 0.519147] usbcore: registered new device driver usb [ 0.522209] pps_core: LinuxPPS API ver. 1 registered [ 0.525016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.529077] PTP clock support registered [ 0.532157] EDAC MC: Ver: 3.0.0 [ 0.535111] PCI: Using ACPI for IRQ routing [ 0.537552] NetLabel: Initializing [ 0.539015] NetLabel: domain hash size = 128 [ 0.541011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543098] NetLabel: unlabeled traffic allowed by default [ 0.545160] vgaarb: loaded [ 0.547278] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.549016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.555877] clocksource: Switched to clocksource kvm-clock [ 0.674773] VFS: Disk quotas dquot_6.6.0 [ 0.676455] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.678711] *** VALIDATE ramfs *** [ 0.680130] *** VALIDATE hugetlbfs *** [ 0.682636] pnp: PnP ACPI init [ 0.685243] pnp: PnP ACPI: found 6 devices [ 0.702990] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.707831] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.711300] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.714750] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.718758] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.723029] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.727699] NET: Registered protocol family 2 [ 0.731480] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.736511] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.740515] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.746763] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.750560] TCP: Hash tables configured (established 65536 bind 65536) [ 0.753911] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.757460] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760373] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.763881] NET: Registered protocol family 1 [ 0.767820] RPC: Registered named UNIX socket transport module. [ 0.771688] RPC: Registered udp transport module. [ 0.775057] RPC: Registered tcp transport module. [ 0.778185] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.782549] NET: Registered protocol family 44 [ 0.784246] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.786942] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.789404] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.792087] PCI: CLS 0 bytes, default 64 [ 0.794130] Unpacking initramfs... [ 2.172282] debug: unmapping init [mem 0xffffa04e7cc64000-0xffffa04e7ffcffff] [ 2.176513] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.178933] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.182192] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.678672] Initialise system trusted keyrings [ 2.679955] Key type blacklist registered [ 2.681220] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.687436] zbud: loaded [ 2.690517] *** VALIDATE nfs *** [ 2.691919] *** VALIDATE nfs4 *** [ 2.693573] pstore: using deflate compression [ 2.697642] Platform Keyring initialized [ 2.796219] NET: Registered protocol family 38 [ 2.797597] Key type asymmetric registered [ 2.798699] Asymmetric key parser 'x509' registered [ 2.800760] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.804625] io scheduler mq-deadline registered [ 2.806415] io scheduler kyber registered [ 2.808550] io scheduler bfq registered [ 2.810854] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.814510] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.817752] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.821398] ACPI: Power Button [PWRF] [ 2.827145] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.834510] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.843328] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.871607] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.898643] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.904175] Non-volatile memory driver v1.3 [ 2.906166] Linux agpgart interface v0.103 [ 2.940025] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.943083] vda: detected capacity change from 0 to 76685312 [ 2.966773] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.969580] vdb: detected capacity change from 0 to 1073741824 [ 2.978616] libphy: Fixed MDIO Bus: probed [ 2.988557] usbcore: registered new interface driver usbserial_generic [ 2.990530] usbserial: USB Serial support registered for generic [ 2.992738] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.998571] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.999796] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.001541] mousedev: PS/2 mouse device common for all mice [ 3.004211] rtc_cmos 00:05: RTC can wake from S4 [ 3.007070] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.007361] rtc_cmos 00:05: registered as rtc0 [ 3.011036] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.014165] intel_pstate: CPU model not supported [ 3.016498] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.019623] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.019666] hid: raw HID events driver (C) Jiri Kosina [ 3.023836] usbcore: registered new interface driver usbhid [ 3.026100] usbhid: USB HID core driver [ 3.027543] drop_monitor: Initializing network drop monitor service [ 3.029642] Initializing XFRM netlink socket [ 3.031946] NET: Registered protocol family 10 [ 3.035757] Segment Routing with IPv6 [ 3.037385] NET: Registered protocol family 17 [ 3.039552] mpls_gso: MPLS GSO support [ 3.045774] RAS: Correctable Errors collector initialized. [ 3.047899] AVX version of gcm_enc/dec engaged. [ 3.049519] AES CTR mode by8 optimization enabled [ 3.133393] sched_clock: Marking stable (3133369914, 0)->(3950804503, -817434589) [ 3.136191] registered taskstats version 1 [ 3.137987] Loading compiled-in X.509 certificates [ 3.139685] zswap: loaded using pool lzo/zbud [ 3.164580] Key type big_key registered [ 3.177710] Key type encrypted registered [ 3.179801] ima: No TPM chip found, activating TPM-bypass! [ 3.182096] ima: Allocated hash algorithm: sha1 [ 3.183673] ima: No architecture policies found [ 3.185443] evm: Initialising EVM extended attributes: [ 3.187345] evm: security.selinux [ 3.188527] evm: security.ima [ 3.189550] evm: security.capability [ 3.190832] evm: HMAC attrs: 0x1 [ 3.193266] rtc_cmos 00:05: setting system clock to 2026-09-05 05:19:20 UTC (1788585560) [ 3.199774] debug: unmapping init [mem 0xffffffff95c03000-0xffffffff95dfffff] [ 3.203142] debug: unmapping init [mem 0xffffffff94982000-0xffffffff94c58fff] [ 3.212105] Write protecting the kernel read-only data: 28672k [ 3.215841] debug: unmapping init [mem 0xffffffff93003000-0xffffffff931fffff] [ 3.218905] debug: unmapping init [mem 0xffffffff93914000-0xffffffff939fffff] [ 3.252001] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.260288] systemd[1]: Detected virtualization kvm. [ 3.262164] systemd[1]: Detected architecture x86-64. [ 3.263913] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.286554] systemd[1]: No hostname configured. [ 3.288242] systemd[1]: Set hostname to . [ 3.291676] random: systemd: uninitialized urandom read (16 bytes read) [ 3.293827] systemd[1]: Initializing machine ID from random generator. [ 3.333196] random: ln: uninitialized urandom read (6 bytes read) [ 3.423431] random: systemd: uninitialized urandom read (16 bytes read) [ 3.426341] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.436790] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.446202] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.971703] device-mapper: uevent: version 1.0.3 [ 3.973724] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.664417] virtio_net virtio0 ens2: renamed from eth0 [ 4.679414] scsi host0: ata_piix [ 4.685964] scsi host1: ata_piix [ 4.688541] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.692743] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.236643] random: fast init done [ 8.589331] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.696079] random: crng init done [ 9.697698] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.127374] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.363551] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.621978] SELinux: Disabled at runtime. [ 11.683849] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.695443] systemd[1]: Detected virtualization kvm. [ 11.697983] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.152124] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.156376] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.163649] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.168316] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.172475] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.180076] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.185453] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ 12.260217] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.712941] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.064080] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.182535] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.221126] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.235531] EDAC sbridge: Ver: 1.1.2 [ 14.488638] Key type dns_resolver registered [ 14.798536] NFS: Registering the id_resolver key type [ 14.800409] Key type id_resolver registered [ 14.802298] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg102-client login: [ 41.491692] libcfs: loading out-of-tree module taints kernel. [ 41.560182] Key type ._llcrypt registered [ 41.561869] Key type .llcrypt registered [ 41.860575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.869227] alg: No test for adler32 (adler32-zlib) [ 42.911608] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 43.254163] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 44.903272] Key type lgssc registered [ 45.662100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.918218] hrtimer: interrupt took 2235241 ns [ 176.668802] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0 [ 182.181587] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 201.197928] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.208686] Lustre: lustre-OST0000-osc-ffffa04ee00f7000: disconnect after 23s idle [ 207.068930] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing yml_node [ 211.734224] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 214.556451] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 217.083695] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 218.895834] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 01:22:54 EDT 2026 [ 235.772858] Lustre: Unmounted lustre-client [ 264.066932] Key type lgssc unregistered [ 264.265882] LNet: 8428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.272920] LNetError: 8428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 264.285681] LNet: Removed LNI 192.168.201.2@tcp [ 264.940313] Key type .llcrypt unregistered [ 264.944195] Key type ._llcrypt unregistered [ 295.595105] Key type ._llcrypt registered [ 295.601147] Key type .llcrypt registered [ 296.090851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 296.108569] alg: No test for adler32 (adler32-zlib) [ 297.001100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 297.075185] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 297.085758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 302.036970] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 302.148417] LNet: 9301:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.169614] LNetError: 9301:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.203620] LNet: Removed LNI 192.168.201.2@tcp [ 302.919494] Key type .llcrypt unregistered [ 302.921348] Key type ._llcrypt unregistered [ 309.858882] Key type ._llcrypt registered [ 309.861865] Key type .llcrypt registered [ 311.418578] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:24:26 (1788585866) [ 311.819352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 311.832737] alg: No test for adler32 (adler32-zlib) [ 312.760855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.832215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.848219] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.864469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.947141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 312.966506] LNet: 9723:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 320.363532] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:24:36 (1788585876) [ 320.505400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.541566] LNet: 10297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.619120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.746799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 329.276376] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:24:44 (1788585884) [ 329.435808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.479341] LNet: 10880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.516930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.691668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 337.944769] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:24:53 (1788585893) [ 338.058742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.104834] LNet: 11465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.129454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.245832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 348.397260] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:25:03 (1788585903) [ 348.547267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.597584] LNet: 12048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.648168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.819503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 349.397630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 356.079570] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:25:11 (1788585911) [ 356.209588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.284175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.373155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 356.665982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 357.001739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 364.627195] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:25:20 (1788585920) [ 364.698662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.737260] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.739801] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 364.763179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.872605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 371.711368] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:25:27 (1788585927) [ 371.833940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.913849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.044330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 372.085154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 372.522834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 372.568169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 373.286502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 373.357502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 373.886480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 373.939366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 374.568257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 374.610682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.156673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 375.209809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 375.866240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 375.895900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 376.423349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 376.450870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 376.929240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 376.967412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 377.430551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 377.470992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.129931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 378.205330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 378.664085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 378.735720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.419988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.471563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.561935] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:25:42 (1788585942) [ 386.627861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.661333] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.667870] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 386.702468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.813585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 386.866845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 394.229762] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:25:50 (1788585950) [ 394.500248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.590243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.660474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.689496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 401.281979] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:25:57 (1788585957) [ 401.516150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.582961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.697538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 401.740858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 410.070198] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:26:05 (1788585965) [ 410.354387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.391476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.505023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 410.591429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 418.421594] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:26:13 (1788585973) [ 418.584273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.682338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.864709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 418.960156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 428.096393] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:26:23 (1788585983) [ 428.366461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.421739] LNet: 17443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.440693] LNet: 17443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 428.508572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.590881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 428.643245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 437.020283] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:26:32 (1788585992) [ 437.386523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.437356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.555718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.180143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.742949] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:26:41 (1788586001) [ 446.014516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.073410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.308568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.642437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.755876] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:26:50 (1788586010) [ 454.890795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.967827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.141483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.321350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.323382] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:26:58 (1788586018) [ 462.473705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.538552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.055685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.443315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.559975] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:27:07 (1788586027) [ 472.677558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.798902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.945518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.019737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.846657] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:27:17 (1788586037) [ 481.956181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.008047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.116650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.175440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 489.093369] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:27:25 (1788586045) [ 489.469724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.520812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.632202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 497.429235] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:27:33 (1788586053) [ 497.683797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.757039] LNet: 22298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.772987] LNet: 22298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 497.815657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.941111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 505.657743] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:27:41 (1788586061) [ 505.912351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.020502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.088768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 506.587392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 514.384448] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:27:50 (1788586070) [ 514.514440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.584121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.712617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 514.736669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 522.290668] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:27:58 (1788586078) [ 522.506710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.536746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.603960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 522.670350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 530.063234] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:28:05 (1788586085) [ 530.177590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.210525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.290872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 530.320274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 538.555554] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:28:14 (1788586094) [ 538.789961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.821455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.948187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 539.026268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 546.159568] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:28:22 (1788586102) [ 546.469411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.504463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.561308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 546.653409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 546.669727] LNetError: 25833:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 546.732425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 546.785219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 553.863519] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:28:29 (1788586109) [ 554.109539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.209294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.283707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.c8zkMxtfka [ 554.400996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.c8zkMxtfka [ 554.477223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.c8zkMxtfka [ 562.055929] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:28:37 (1788586117) [ 562.185539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.279030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.379914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 562.448216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 562.510547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 569.538994] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:28:45 (1788586125) [ 570.048508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.077615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.140589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 570.165762] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 571.623705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 573.009505] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 574.912547] Key type lgssc registered [ 576.066029] Lustre: Echo OBD driver; http://www.lustre.org/ [ 579.468352] LNetError: 27595:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.2@tcp99 has no available nets [ 581.870278] Key type lgssc unregistered [ 582.276918] LNetError: 28700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.293335] LNet: Removed LNI 192.168.201.2@tcp [ 583.071797] Key type .llcrypt unregistered [ 583.079239] Key type ._llcrypt unregistered [ 589.319411] Key type ._llcrypt registered [ 589.321360] Key type .llcrypt registered [ 590.738280] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:29:06 (1788586146) [ 591.466799] Key type .llcrypt unregistered [ 591.468895] Key type ._llcrypt unregistered [ 599.499405] Key type ._llcrypt registered [ 599.502380] Key type .llcrypt registered [ 599.743264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 599.759666] alg: No test for adler32 (adler32-zlib) [ 600.727545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.847115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 600.890442] LNet: 29611:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 600.902420] LNet: 29611:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 600.909178] LNet: 29611:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 600.927779] LNet: Message forwarding will be enabled in 2s [ 603.999130] LNet: Message forwarding enabled [ 604.532665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.579423] LNet: Message forwarding will stop in 2s to 6s [ 608.103883] LNet: Message forwarding disabled [ 609.203852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 609.232435] LNet: 29709:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 609.242602] LNet: 29709:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 609.249550] LNet: 29709:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 609.266622] LNet: Message forwarding will be enabled in 2s [ 609.445234] LNet: 29722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.969473] Key type .llcrypt unregistered [ 609.972601] Key type ._llcrypt unregistered [ 622.785563] Key type ._llcrypt registered [ 622.790367] Key type .llcrypt registered [ 624.326328] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:29:39 (1788586179) [ 624.821643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.835257] alg: No test for adler32 (adler32-zlib) [ 625.744712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.810878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 632.523550] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:29:48 (1788586188) [ 632.808880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.831313] LNet: 31121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.855929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.913951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 632.947924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 633.011962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 633.049109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 633.079740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 633.121173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 633.163955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 633.229819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 633.275821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 633.325922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 633.366705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 633.419932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 633.469587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 633.526982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 633.583385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 633.640472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 633.707567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 633.809770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 633.869636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 633.900449] LNetError: 31164:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 640.999524] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:29:56 (1788586196) [ 641.168739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.210586] LNet: 31734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.271110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.373596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.128416] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:30:04 (1788586204) [ 651.803819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg102-server [ 653.820972] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:30:09 (1788586209) [ 654.157164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.196632] LNet: 32666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.236266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.299097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.318607] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 655.239682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[3-7]@tcp [ 655.264504] LNet: 32800:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 655.326179] LNetError: 32670:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.4@tcp is being used as a gateway but routing feature is not turned on [ 655.350416] LNetError: 32670:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.4@tcp has gone from up to down [ 662.971969] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:30:18 (1788586218) [ 665.516456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg102-server [ 667.833333] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:30:23 (1788586223) [ 667.999155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.067248] LNet: 33730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.088039] LNetError: 33730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.128887] LNet: Removed LNI 192.168.201.2@tcp [ 668.211588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.289620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.330516] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 669.356749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[3-8/2]@tcp [ 669.389783] LNet: 33853:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 669.403681] LNet: 33853:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 669.446332] LNetError: 33740:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.5@tcp is being used as a gateway but routing feature is not turned on [ 669.453961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[3-8/2]@tcp [ 669.472321] LNetError: 33740:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 669.472348] LNetError: 33740:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.5@tcp has gone from up to down [ 669.472352] LNetError: 33740:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 676.963876] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:30:32 (1788586232) [ 677.263720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.298567] LNet: 34446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.305028] LNetError: 34446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.323213] LNet: Removed LNI 192.168.201.2@tcp [ 677.355401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.502610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 677.586687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.621738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.688086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.779904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.828899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.915135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 678.029813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.098303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.224742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.318237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.361267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.458430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 678.545022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.598984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.687915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 678.796310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.847058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.930996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 679.099931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.175246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.317386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.421571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.460966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.547797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 679.637490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.688160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.767921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.863273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.892726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.957789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 680.007714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.036099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.091526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.327962] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:30:42 (1788586242) [ 688.995933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg102-server [ 690.812540] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:30:46 (1788586246) [ 692.823210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg102-server [ 694.716566] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:30:50 (1788586250) [ 695.264440] LNet: 35849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.272487] LNet: 35849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 695.962531] Key type .llcrypt unregistered [ 695.964824] Key type ._llcrypt unregistered [ 703.363196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 703.595151] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 703.607457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 703.616476] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 706.263542] Key type ._llcrypt registered [ 706.266140] Key type .llcrypt registered [ 706.426657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.450164] alg: No test for adler32 (adler32-zlib) [ 707.407470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.506385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.537250] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 708.194404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.215610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 710.246519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 710.265499] LNet: Removed LNI 10.1.2.3@tcp [ 710.447927] LNet: 36563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.456084] LNetError: 36563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.480737] LNet: Removed LNI 192.168.201.2@tcp [ 711.244704] Key type .llcrypt unregistered [ 711.246826] Key type ._llcrypt unregistered [ 726.096427] Key type ._llcrypt registered [ 726.107672] Key type .llcrypt registered [ 727.474392] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:31:23 (1788586283) [ 728.041541] Key type .llcrypt unregistered [ 728.042737] Key type ._llcrypt unregistered [ 736.412067] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 736.681901] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 736.694883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 736.712187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 739.649478] Key type ._llcrypt registered [ 739.653030] Key type .llcrypt registered [ 739.778957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.791159] alg: No test for adler32 (adler32-zlib) [ 740.711716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.797739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.811474] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 741.593839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.628284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 744.735806] LNet: 38050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.748471] LNetError: 38050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 744.755617] LNet: Removed LNI 10.1.2.3@tcp [ 744.770498] LNet: Removed LNI 192.168.201.2@tcp [ 745.534485] Key type .llcrypt unregistered [ 745.535996] Key type ._llcrypt unregistered [ 762.086855] Key type ._llcrypt registered [ 762.088908] Key type .llcrypt registered [ 764.096544] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:31:59 (1788586319) [ 764.852446] Key type .llcrypt unregistered [ 764.862528] Key type ._llcrypt unregistered [ 774.478721] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.707043] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.709561] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.712290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.319520] Key type ._llcrypt registered [ 777.327214] Key type .llcrypt registered [ 777.505947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.519753] alg: No test for adler32 (adler32-zlib) [ 778.505826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.630906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 778.674594] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 781.447913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.464951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.934720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 783.957905] LNet: Removed LNI 10.1.2.3@tcp [ 783.976717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 784.015533] LNet: Removed LNI 10.1.2.31@tcp [ 784.036927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.065988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.935671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 784.960949] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 785.820882] LNet: 39965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.830864] LNetError: 39965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 785.840338] LNet: Removed LNI 10.1.2.31@tcp [ 785.857665] LNet: Removed LNI 10.1.2.3@tcp [ 786.853854] Key type .llcrypt unregistered [ 786.856287] Key type ._llcrypt unregistered [ 800.151941] Key type ._llcrypt registered [ 800.155858] Key type .llcrypt registered [ 802.168696] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:32:37 (1788586357) [ 802.595662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 802.627759] alg: No test for adler32 (adler32-zlib) [ 803.718648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.879566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.931835] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 805.513687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.536842] LNet: 40916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.551638] LNetError: 40916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.569973] LNet: Removed LNI 192.168.201.2@tcp [ 805.621308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.742452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.2@tcp [ 805.776299] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 807.273281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.2@tcp [ 807.308866] LNet: Removed LNI 192.168.201.2@tcp [ 807.432678] LNet: 41051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.244763] Key type .llcrypt unregistered [ 808.250995] Key type ._llcrypt unregistered [ 824.411975] Key type ._llcrypt registered [ 824.413142] Key type .llcrypt registered [ 826.602988] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:33:01 (1788586381) [ 829.452977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg102-server [ 831.409963] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:33:07 (1788586387) [ 832.244698] Key type .llcrypt unregistered [ 832.254165] Key type ._llcrypt unregistered [ 840.130145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.427396] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.433757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.470100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 843.414971] Key type ._llcrypt registered [ 843.417962] Key type .llcrypt registered [ 843.623871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.657129] alg: No test for adler32 (adler32-zlib) [ 844.652457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.758728] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 845.660034] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 847.975513] LNet: 42882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.014138] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.027054] LNet: Removed LNI 192.168.201.2@tcp1 [ 848.039658] LNet: Removed LNI 10.1.2.3@tcp2 [ 848.892837] Key type .llcrypt unregistered [ 848.901961] Key type ._llcrypt unregistered [ 864.132359] Key type ._llcrypt registered [ 864.135853] Key type .llcrypt registered [ 865.534954] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:33:41 (1788586421) [ 866.291652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 866.327470] alg: No test for adler32 (adler32-zlib) [ 867.430154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.548228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.574476] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 868.367338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.411780] LNet: 43849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.419880] LNetError: 43849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.444543] LNet: Removed LNI 192.168.201.2@tcp [ 868.492678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.600207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 868.632491] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 876.268226] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:33:52 (1788586432) [ 876.499845] LNet: 44439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.506435] LNetError: 44439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.535620] LNet: Removed LNI 192.168.201.2@tcp [ 877.389617] Key type .llcrypt unregistered [ 877.395311] Key type ._llcrypt unregistered [ 883.934017] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.066784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 884.079847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 884.106599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.952914] Key type ._llcrypt registered [ 886.954511] Key type .llcrypt registered [ 887.196374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.220881] alg: No test for adler32 (adler32-zlib) [ 888.275474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.349538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.400519] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 889.570537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 889.651746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.489231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.529962] LNet: 45180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.538725] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 892.547858] LNet: Removed LNI 10.1.2.3@tcp [ 892.568339] LNet: Removed LNI 192.168.201.2@tcp [ 892.593560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.685980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 892.725630] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 892.734724] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.431284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.486919] LNet: 45207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.498691] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.508365] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 893.523501] LNet: Removed LNI 10.1.2.3@tcp [ 893.541060] LNet: Removed LNI 192.168.201.2@tcp [ 893.588850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.703909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.770356] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 893.785630] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.353124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 894.377574] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 895.489398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.522717] LNet: 45369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.537424] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.548462] LNetError: 45369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 895.559683] LNet: Removed LNI 10.1.2.3@tcp [ 895.561374] LNet: Removed LNI 192.168.201.2@tcp [ 895.588486] LNet: Removed LNI 192.168.201.2@tcp2 [ 895.628153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.722354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 895.749031] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 895.759082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.768318] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 904.586415] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:34:19 (1788586459) [ 904.682161] LNet: 45961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.687850] LNetError: 45961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 904.699690] LNetError: 45961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 904.710754] LNet: Removed LNI 10.1.2.3@tcp [ 904.712214] LNet: Removed LNI 192.168.201.2@tcp [ 904.729775] LNet: Removed LNI 192.168.201.2@tcp2 [ 905.602111] Key type .llcrypt unregistered [ 905.603910] Key type ._llcrypt unregistered [ 913.003715] Key type ._llcrypt registered [ 913.007747] Key type .llcrypt registered [ 913.229116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.272524] alg: No test for adler32 (adler32-zlib) [ 914.283879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.351676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.385223] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 915.394777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.416746] LNet: 46361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.429688] LNetError: 46361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.441972] LNet: Removed LNI 192.168.201.2@tcp [ 915.463280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.531043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.561643] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 916.137967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.165969] LNet: 46391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.174652] LNetError: 46391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.197154] LNet: Removed LNI 192.168.201.2@tcp [ 916.238970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.355321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.409748] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 916.906306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.934939] LNet: Removed LNI 192.168.201.2@tcp [ 916.953387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.011122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.041930] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 917.557536] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.809396] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.815227] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.824522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.263885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.301460] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.310687] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 920.317956] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.325988] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 920.353472] LNet: Removed LNI 192.168.201.2@tcp [ 920.389841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.535236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 920.662926] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 928.158947] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:34:43 (1788586483) [ 928.304812] LNet: 47083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.323555] LNetError: 47083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.355506] LNet: Removed LNI 192.168.201.2@tcp [ 929.170285] Key type .llcrypt unregistered [ 929.176704] Key type ._llcrypt unregistered [ 938.531293] Key type ._llcrypt registered [ 938.540058] Key type .llcrypt registered [ 938.784420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.841673] alg: No test for adler32 (adler32-zlib) [ 939.773434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.821605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.853635] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 940.660192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.676577] LNet: 47487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.680751] LNetError: 47487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.696292] LNet: Removed LNI 192.168.201.2@tcp [ 940.711512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.771602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 940.790823] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 948.523400] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:35:03 (1788586503) [ 948.823792] LNet: 48079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.828979] LNetError: 48079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.857479] LNet: Removed LNI 192.168.201.2@tcp [ 949.697518] Key type .llcrypt unregistered [ 949.701584] Key type ._llcrypt unregistered [ 957.388265] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 957.666585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 957.679291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 957.687407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 960.600769] Key type ._llcrypt registered [ 960.605694] Key type .llcrypt registered [ 960.929494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.963365] alg: No test for adler32 (adler32-zlib) [ 962.047760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.142634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.176217] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 963.335992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.369963] LNet: 48526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.406780] LNetError: 48526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.437110] LNet: Removed LNI 192.168.201.2@tcp [ 963.484895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.546613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 963.599185] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 970.971773] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:35:27 (1788586527) [ 971.339186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.367755] LNet: 49124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.380579] LNetError: 49124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.398311] LNet: Removed LNI 192.168.201.2@tcp [ 971.433721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.581970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 977.875662] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:35:33 (1788586533) [ 978.378641] LNet: 49707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.038616] Key type .llcrypt unregistered [ 979.041727] Key type ._llcrypt unregistered [ 986.180625] Key type ._llcrypt registered [ 986.182507] Key type .llcrypt registered [ 986.466566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 986.478908] alg: No test for adler32 (adler32-zlib) [ 987.365164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.416430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 987.436541] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 988.396910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.449409] LNet: 50101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.457679] LNetError: 50101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.474435] LNet: Removed LNI 192.168.201.2@tcp [ 988.523182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.696535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 988.740384] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 997.444437] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:35:52 (1788586552) [ 997.591855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 997.639873] LNet: 50687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.652795] LNetError: 50687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.672212] LNet: Removed LNI 192.168.201.2@tcp [ 997.740694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.818354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1004.990650] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:36:00 (1788586560) [ 1006.532899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1008.452123] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:36:04 (1788586564) [ 1010.162285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1011.744792] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:36:07 (1788586567) [ 1013.799847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1015.942723] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:36:11 (1788586571) [ 1017.854343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1019.717975] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:36:15 (1788586575) [ 1021.746768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1023.714427] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:36:19 (1788586579) [ 1023.921989] LNet: 52744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.581074] Key type .llcrypt unregistered [ 1024.586731] Key type ._llcrypt unregistered [ 1032.105093] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.346445] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.358956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.372888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.176734] Key type ._llcrypt registered [ 1035.180539] Key type .llcrypt registered [ 1035.406472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.426838] alg: No test for adler32 (adler32-zlib) [ 1036.371353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.458237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.476057] LNet: Added LNI 192.168.201.2@tcp [8/512/0/180] [ 1036.480766] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1036.484424] LNet: 53069:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1036.604126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.639968] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.645996] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.658966] LNet: Removed LNI 10.1.2.3@tcp [ 1036.662378] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.675671] LNet: Removed LNI 192.168.201.2@tcp [ 1036.717131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.815600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.844280] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1036.949808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.983204] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.989703] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.004655] LNet: Removed LNI 192.168.201.2@tcp [ 1037.041654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.133447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.163411] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1037.168556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.170939] LNet: 53120:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.252957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.272401] LNet: 53138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.280677] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.284767] LNet: Removed LNI 10.1.2.3@tcp [ 1037.288992] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.302847] LNet: Removed LNI 192.168.201.2@tcp [ 1037.336420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.415819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.444922] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1037.448776] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.451690] LNet: 53146:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.629380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.651113] LNet: 53172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.655521] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.660733] LNet: Removed LNI 10.1.2.3@tcp [ 1037.662330] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.680898] LNet: Removed LNI 192.168.201.2@tcp [ 1037.701744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.797785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.848328] LNet: Added LNI 192.168.201.2@tcp [8/256/0/182] [ 1037.853807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1037.998792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.038836] LNet: 53200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.047426] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.055426] LNet: Removed LNI 10.1.2.3@tcp [ 1038.060778] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.092164] LNet: Removed LNI 192.168.201.2@tcp [ 1038.127538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.242702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.291997] LNet: Added LNI 192.168.201.2@tcp [8/256/0/182] [ 1038.306425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.311315] LNet: 53208:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.403421] LNet: Removed LNI 10.1.2.3@tcp [ 1038.418065] LNet: Removed LNI 192.168.201.2@tcp [ 1039.039327] Key type .llcrypt unregistered [ 1039.044706] Key type ._llcrypt unregistered [ 1052.227113] Key type ._llcrypt registered [ 1052.229235] Key type .llcrypt registered [ 1054.180299] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:36:49 (1788586609) [ 1056.489821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1058.928349] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:36:54 (1788586614) [ 1059.544368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.577613] alg: No test for adler32 (adler32-zlib) [ 1060.599644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1060.680562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1060.739132] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1061.848378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.2 [ 1061.871817] LNet: 54470:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1061.881878] LNet: Removed LNI 192.168.201.2@tcp [ 1069.318362] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:37:04 (1788586624) [ 1069.894585] LNet: 55040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.903077] LNetError: 55040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.918866] LNet: Removed LNI 192.168.201.2@tcp [ 1070.489323] Key type .llcrypt unregistered [ 1070.491066] Key type ._llcrypt unregistered [ 1076.932744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.247718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.108430] Key type ._llcrypt registered [ 1079.114165] Key type .llcrypt registered [ 1079.291613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.319184] alg: No test for adler32 (adler32-zlib) [ 1080.682391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1087.580874] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:37:23 (1788586643) [ 1087.661397] LNet: 55932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.667442] LNetError: 55932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1087.682152] LNet: Removed LNI 10.1.2.3@tcp [ 1088.426746] Key type .llcrypt unregistered [ 1088.429926] Key type ._llcrypt unregistered [ 1097.345633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1097.611445] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1097.622141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1097.634363] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1099.470069] Key type ._llcrypt registered [ 1099.484965] Key type .llcrypt registered [ 1099.845681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1099.869616] alg: No test for adler32 (adler32-zlib) [ 1101.094216] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1111.414477] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:37:46 (1788586666) [ 1113.268066] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1115.889692] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:37:51 (1788586671) [ 1116.111405] LNet: 57109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.119668] LNetError: 57109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.159668] LNet: Removed LNI 10.1.2.3@tcp [ 1117.593972] Key type .llcrypt unregistered [ 1117.597496] Key type ._llcrypt unregistered [ 1126.109289] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1126.407672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1126.412847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1126.420366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1128.321907] Key type ._llcrypt registered [ 1128.323917] Key type .llcrypt registered [ 1128.657701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1128.701332] alg: No test for adler32 (adler32-zlib) [ 1129.752604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1129.857857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1140.562123] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:38:16 (1788586696) [ 1141.506109] LNet: 58335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.530747] LNetError: 58335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1141.577305] LNet: Removed LNI 10.1.2.3@tcp [ 1142.659555] Key type .llcrypt unregistered [ 1142.662926] Key type ._llcrypt unregistered [ 1150.430747] Key type ._llcrypt registered [ 1150.434805] Key type .llcrypt registered [ 1150.804586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.838982] alg: No test for adler32 (adler32-zlib) [ 1151.785403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1151.821841] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1159.944648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1160.814477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1160.858603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1160.922483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1160.957876] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1161.055605] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1161.349188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.392741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.430577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.487506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.536037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1161.586871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.621641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.665507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.707376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.726783] LNet: 58772:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.747078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.775536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.809328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.843691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.884950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.919492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1161.994427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.819204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1162.864966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1162.903259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1162.921588] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1162.928343] LNet: Skipped 1 previous similar message [ 1163.214870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.252942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.293326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.327431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.361738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1163.397747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.430848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.462364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.497425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.516744] LNet: 58883:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.537948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.569639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.590640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.608068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.622443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.638589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.669885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.212166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1164.257505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1164.338028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1164.378063] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1164.725470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.759683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.832831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.863366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.892611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1164.924486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.956914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.995622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.031400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.048423] LNet: 58995:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.074816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.110582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.144825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.173446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.210833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.252652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.287800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.845613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1165.895685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1165.947874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1166.246958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.283204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.312903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.335841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.370731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1166.400400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.436419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.464561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.516905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.571095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.619415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.658549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.747655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.828223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.904462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.966609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1167.735572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1167.822977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1167.901661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1167.943841] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1167.958699] LNet: Skipped 2 previous similar messages [ 1168.220894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.256913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.301957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.334353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.363342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1168.396886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.432794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.462292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.491528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.510473] LNet: 59217:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.516252] LNet: 59217:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1168.545507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.592695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.633786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.668567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.754579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.832734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.894128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.639628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1169.720597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1169.789412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1170.082911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.105770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.136793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.160894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.184622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1170.213902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.238876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.270711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.311388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.337583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.370805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.449929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.486935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.518996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.573499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.632294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.093315] LNet: 59369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.102870] LNetError: 59369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.135195] LNet: Removed LNI 192.168.201.2@tcp [ 1172.672641] Key type .llcrypt unregistered [ 1172.679599] Key type ._llcrypt unregistered [ 1194.813062] Key type ._llcrypt registered [ 1194.820628] Key type .llcrypt registered [ 1196.324771] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:39:11 (1788586751) [ 1197.649714] Key type .llcrypt unregistered [ 1197.653760] Key type ._llcrypt unregistered [ 1206.616926] Key type ._llcrypt registered [ 1206.620611] Key type .llcrypt registered [ 1206.962294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.000942] alg: No test for adler32 (adler32-zlib) [ 1208.004559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.056189] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1217.583952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1230.664101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.715492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1230.752989] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1232.330090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1232.396699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 1 [ 1232.477276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1232.583833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 1 [ 1232.661613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1232.746408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 1 [ 1232.814415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1232.910740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 1 [ 1232.981825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1233.039471] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1233.051505] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1233.482931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.523336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1233.579217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.625043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.682415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1233.761370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.830547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.874485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.902789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.955362] LNet: 60838:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.986287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.038214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.074632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.080904] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1234.118923] LNet: Skipped 3 previous similar messages [ 1234.159239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.199566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.260742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.317719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.145946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1235.189595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 1 [ 1235.236695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1235.316305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 1 [ 1235.368625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1235.413331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 1 [ 1235.467795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1235.512817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 1 [ 1235.558450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1235.581823] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1235.869505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.938501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.003546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.065286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.107432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1236.177599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.207789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.255837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.309439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.349797] LNet: 60961:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.381531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.451773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.517903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.575415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.651723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.725605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.792585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.581163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1237.623918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 1 [ 1237.675606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1237.720328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 1 [ 1237.769248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1237.851592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 1 [ 1237.936233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1237.999745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 1 [ 1238.053372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1238.074947] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1238.083470] LNet: Skipped 2 previous similar messages [ 1238.340511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.378750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.410266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.441078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.473283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1238.533049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.585154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.633120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.676301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.705806] LNet: 61084:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.734967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.764577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.811557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.875885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.911562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.933513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.974945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.566441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1239.608133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 1 [ 1239.641304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1239.691425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 1 [ 1239.736394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1239.765900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 1 [ 1239.805541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1239.847649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 1 [ 1239.891403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1240.174910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.206205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.224960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.243357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.265315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1240.285832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.314119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.338448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.361294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.377543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.395818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.420110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.450723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.479098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.505434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.541324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1241.179557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1241.212234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 1 [ 1241.231544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1241.254395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 1 [ 1241.304466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1241.362634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 1 [ 1241.396334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1241.426886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 1 [ 1241.467091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1241.740571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.783985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.812717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.844073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.875821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.911738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.937198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.988317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.042133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.069036] LNet: 61331:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.074662] LNet: 61331:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1242.093966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.153319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.211305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.246698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.279884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.309915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.347143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.923191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1242.959685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_error -r 1 [ 1242.996445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1243.019608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_error -r 1 [ 1243.053258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1243.079321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_error -r 1 [ 1243.100069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1243.142350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_error -r 1 [ 1243.162309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1243.179745] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1243.184774] LNet: Skipped 8 previous similar messages [ 1243.426126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.467374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.500563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.550785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.598110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.689387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.733196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.801664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.849109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.881773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.924990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.965731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.006211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.044812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.088567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.110845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1249.634930] LNet: 61530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.665332] LNetError: 61530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.691064] LNet: Removed LNI 192.168.201.2@tcp [ 1250.731961] LNet: Removed LNI 192.168.201.2@tcp1 [ 1251.409925] Key type .llcrypt unregistered [ 1251.416372] Key type ._llcrypt unregistered [ 1273.708661] Key type ._llcrypt registered [ 1273.712613] Key type .llcrypt registered [ 1274.926736] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:40:30 (1788586830) [ 1275.754558] Key type .llcrypt unregistered [ 1275.757942] Key type ._llcrypt unregistered [ 1283.144905] Key type ._llcrypt registered [ 1283.148144] Key type .llcrypt registered [ 1283.380783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.400746] alg: No test for adler32 (adler32-zlib) [ 1284.486140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1284.527039] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1292.821744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1293.717042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1293.758817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1293.831551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1293.858088] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1294.153276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.182419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1294.218204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.244593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.280336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1294.319989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.389607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1294.441663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1294.485973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1294.505631] LNet: 62776:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1294.530086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1294.566764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1294.601270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1294.625779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.659360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1294.691831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1294.724664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.313271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1295.355418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1295.412339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1295.442925] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1295.688149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.717145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.752430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.797207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.844813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1295.887996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.915261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.950123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1296.020078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1296.038786] LNet: 62888:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1296.070519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1296.096279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1296.120601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1296.147635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.172227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.201445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.237647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1296.844148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1296.911953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1296.989668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1297.036519] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1297.303703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.355712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.381276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.414167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.472337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1297.520460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.577450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.633719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.704436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.735339] LNet: 62999:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.762410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.806269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.839582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.876988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.935747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.964631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.999710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1298.329529] LNet: 63040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.335707] LNetError: 63040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.354520] LNet: Removed LNI 192.168.201.2@tcp [ 1298.823837] Key type .llcrypt unregistered [ 1298.829165] Key type ._llcrypt unregistered [ 1315.465696] Key type ._llcrypt registered [ 1315.469570] Key type .llcrypt registered [ 1317.390754] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:41:12 (1788586872) [ 1318.383507] Key type .llcrypt unregistered [ 1318.386470] Key type ._llcrypt unregistered [ 1325.797904] Key type ._llcrypt registered [ 1325.800274] Key type .llcrypt registered [ 1325.933038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1325.942233] alg: No test for adler32 (adler32-zlib) [ 1326.950233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1327.005572] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1335.926395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1345.751605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.814217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1345.836972] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1347.654522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1347.692518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 1 [ 1347.735354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1347.798380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 1 [ 1347.860809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1347.977764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 1 [ 1348.072659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1348.131195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 1 [ 1348.204663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1348.256065] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1348.263034] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 1348.654126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.703687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.765369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.809149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.854760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1348.892397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.937419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.993806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.032736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.079675] LNet: 64503:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.109883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.186563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.255103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.291360] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1349.307316] LNet: Skipped 3 previous similar messages [ 1349.326725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.368028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.429978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.474743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1349.541389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.384541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1350.421659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_error -r 1 [ 1350.461390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1350.496569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_error -r 1 [ 1350.533390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1350.578947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_error -r 1 [ 1350.621552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1350.676712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_error -r 1 [ 1350.728172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1350.751941] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1351.177678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1351.235559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1351.273656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1351.306385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1351.369077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1351.455091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.546517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.607243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.649202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.667922] LNet: 64629:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1351.709138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.768224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.842625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.879627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.954627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.987286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1352.031182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1352.116250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.038286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1353.098259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_timeout -r 1 [ 1353.153273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1353.228449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_timeout -r 1 [ 1353.301634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1353.398141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_timeout -r 1 [ 1353.452127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1353.532767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_timeout -r 1 [ 1353.609972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1353.636616] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1353.897824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.941688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.969218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.018282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.068412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1354.110427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.149535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.193480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.234339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.259927] LNet: 64754:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1354.301056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.378698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.451312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.498704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.558471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.631679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.673686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1354.725470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.084638] LNet: 64832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.093833] LNetError: 64832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.105763] LNet: Removed LNI 192.168.201.2@tcp [ 1360.136239] LNet: Removed LNI 192.168.201.2@tcp1 [ 1360.760921] Key type .llcrypt unregistered [ 1360.766128] Key type ._llcrypt unregistered [ 1379.325853] Key type ._llcrypt registered [ 1379.329863] Key type .llcrypt registered [ 1380.817736] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:42:16 (1788586936) [ 1382.560750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1384.358519] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:42:20 (1788586940) [ 1385.216627] Key type .llcrypt unregistered [ 1385.222046] Key type ._llcrypt unregistered [ 1390.832846] Key type ._llcrypt registered [ 1390.835715] Key type .llcrypt registered [ 1391.107788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1391.129141] alg: No test for adler32 (adler32-zlib) [ 1392.016711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1392.037376] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1399.644620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1400.631172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1400.699548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1400.750059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1400.778605] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1401.074732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1401.126646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1401.151302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1401.175119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1401.215870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1401.248658] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1401.276690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1401.335590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1401.379036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1401.430897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1401.453386] LNet: 66373:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1401.473867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1401.509668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1401.537550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.563954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.605439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1401.660919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.952652] LNet: 66412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.962446] LNetError: 66412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.985224] LNet: Removed LNI 192.168.201.2@tcp [ 1402.563460] Key type .llcrypt unregistered [ 1402.565524] Key type ._llcrypt unregistered [ 1423.471264] Key type ._llcrypt registered [ 1423.478605] Key type .llcrypt registered [ 1423.794617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1423.812861] alg: No test for adler32 (adler32-zlib) [ 1424.787355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1424.819561] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1432.830908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1441.557407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.601912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.635963] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1442.898310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1442.927163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e network_timeout -r 1 [ 1442.984975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1443.071796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e network_timeout -r 1 [ 1443.128431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1443.180883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e network_timeout -r 1 [ 1443.214785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1443.246807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e network_timeout -r 1 [ 1443.277846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1443.308174] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1443.561740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.606286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1443.637598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.684820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.712750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1443.741557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.788891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.822096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.846826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.883842] LNet: 67289:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.928701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1444.007585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1444.052682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.109763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.201800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1444.262658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1449.485383] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1449.893573] LNet: 67362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.900930] LNetError: 67362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.906983] LNet: Removed LNI 192.168.201.2@tcp [ 1449.940450] LNet: Removed LNI 192.168.201.2@tcp1 [ 1450.381636] Key type .llcrypt unregistered [ 1450.383712] Key type ._llcrypt unregistered [ 1466.969566] Key type ._llcrypt registered [ 1466.971186] Key type .llcrypt registered [ 1468.544997] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:43:44 (1788587024) [ 1469.017180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1469.032745] alg: No test for adler32 (adler32-zlib) [ 1469.980063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.053812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.082530] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1471.270486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1471.301283] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1472.532900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1472.625783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1472.671423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1472.722983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1472.767357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1472.806735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1472.851335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1472.875244] LNetError: 68460:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1473.119634] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1473.122933] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1475.168348] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1475.559966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1479.263586] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1480.485557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.530433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1480.599297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.627201] LNet: 68567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.634814] LNetError: 68567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.642296] LNet: Removed LNI 192.168.201.2@tcp [ 1480.664836] LNet: Removed LNI 192.168.201.2@tcp1 [ 1480.688395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.775755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.804487] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1481.979736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.997473] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1483.062514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1483.108877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.181838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1483.230333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1483.292346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1483.336193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1483.407684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.464661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1483.513053] LNetError: 68839:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1483.807467] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1486.179256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1489.954127] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1489.961412] LNet: Skipped 1 previous similar message [ 1493.391715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.460848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1493.513946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1500.822308] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:44:16 (1788587056) [ 1501.111797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.131910] LNet: 69537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.136759] LNetError: 69537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.139883] LNetError: 69537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.148586] LNet: Removed LNI 192.168.201.2@tcp [ 1501.161499] LNet: Removed LNI 192.168.201.2@tcp1 [ 1501.188216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.289246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.334476] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1502.332663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.357143] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1503.194591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1503.264714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1503.309786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1503.373172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1503.446443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1503.545471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.628555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 0 [ 1503.801865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.355347] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.2@tcp [ 1504.360514] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1504.360525] LNet: Skipped 1 previous similar message [ 1519.557724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.784183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1519.812370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.853076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 500 [ 1519.963202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.836392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1526.964697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.000075] LNet: 70029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.006747] LNetError: 70029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.013692] LNetError: 70029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1527.021467] LNet: Removed LNI 192.168.201.2@tcp [ 1527.036609] LNet: Removed LNI 192.168.201.2@tcp1 [ 1527.076351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.143248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.180339] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1528.024256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.058276] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1528.903131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1528.937799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1529.016362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1529.049336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1529.092736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1529.164825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1529.235769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1529.290367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 0 [ 1530.210413] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1530.213709] LNet: Skipped 3 previous similar messages [ 1530.743168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.444441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1541.496575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1541.544436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.328288] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:45:04 (1788587104) [ 1549.983364] LNet: 71077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.992390] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.000975] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1550.021087] LNet: Removed LNI 192.168.201.2@tcp [ 1550.038471] LNet: Removed LNI 192.168.201.2@tcp1 [ 1551.137867] Key type .llcrypt unregistered [ 1551.139425] Key type ._llcrypt unregistered [ 1558.351750] Key type ._llcrypt registered [ 1558.353400] Key type .llcrypt registered [ 1558.563829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.587412] alg: No test for adler32 (adler32-zlib) [ 1559.705851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1559.764148] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1571.170406] Lustre: DEBUG MARKER: Initial discovery [ 1571.364295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1574.129995] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1574.213553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1574.283852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1574.333316] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1574.375888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1574.466976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.102@tcp -r 1 -m GET -l 3 [ 1574.495701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.102@tcp -r 1 -m GET -e local_error [ 1574.529305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.102@tcp -r 1 -m PUT -l 6 [ 1575.140386] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1576.084549] Lustre: DEBUG MARKER: Force oleg102-client.virtnet to discover 192.168.201.102@tcp (in background) [ 1576.202861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1577.849234] Lustre: DEBUG MARKER: Force oleg102-server to discover 192.168.201.2@tcp [ 1579.295894] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1580.651692] Lustre: DEBUG MARKER: Wait for 71831 [ 1582.268619] Lustre: DEBUG MARKER: Finished wait on 71831 [ 1582.442434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1582.478463] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1582.488191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1582.731531] LNet: 72287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.736386] LNetError: 72287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.788954] LNet: Removed LNI 192.168.201.2@tcp [ 1584.389734] Key type .llcrypt unregistered [ 1584.395190] Key type ._llcrypt unregistered [ 1600.497404] Key type ._llcrypt registered [ 1600.498913] Key type .llcrypt registered [ 1601.689524] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:45:57 (1788587157) [ 1602.061211] Key type .llcrypt unregistered [ 1602.064891] Key type ._llcrypt unregistered [ 1608.729842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1608.838886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1608.845575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1608.851471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1611.472602] Key type ._llcrypt registered [ 1611.475499] Key type .llcrypt registered [ 1611.650604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.660977] alg: No test for adler32 (adler32-zlib) [ 1612.595211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.647934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.673523] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1613.401994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1613.419184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1622.354157] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:46:18 (1788587178) [ 1622.543449] LNet: 74354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.553228] LNetError: 74354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1622.563952] LNet: Removed LNI 10.1.2.3@tcp [ 1622.577836] LNet: Removed LNI 192.168.201.2@tcp [ 1623.136787] Key type .llcrypt unregistered [ 1623.143807] Key type ._llcrypt unregistered [ 1629.303432] Key type ._llcrypt registered [ 1629.308240] Key type .llcrypt registered [ 1629.461996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.471348] alg: No test for adler32 (adler32-zlib) [ 1630.395437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.447543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.470592] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1631.159250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1631.179488] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1632.057906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.2@tcp [ 1633.124211] LNetError: Unexpected error -22 connecting to 192.168.201.2@tcp at host 192.168.201.2:988 [ 1638.468472] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:46:34 (1788587194) [ 1638.647237] LNet: 75481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.660544] LNetError: 75481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1638.674981] LNet: Removed LNI 10.1.2.3@tcp [ 1638.724572] LNet: Removed LNI 192.168.201.2@tcp [ 1639.454705] Key type .llcrypt unregistered [ 1639.460392] Key type ._llcrypt unregistered [ 1645.581293] Key type ._llcrypt registered [ 1645.587626] Key type .llcrypt registered [ 1645.799596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1645.818539] alg: No test for adler32 (adler32-zlib) [ 1646.778757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.823291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1646.852072] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1647.444924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1647.464035] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 1648.309344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.2@tcp1 --nid 192.168.201.2@tcp2 [ 1652.204727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp1 192.168.201.2@tcp2 [ 1652.222297] LNetError: 76318:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.2@tcp1 to 192.168.201.2@tcp2 - Route table may be misconfigured [ 1652.240976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp2 192.168.201.2@tcp1 [ 1658.415706] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:46:54 (1788587214) [ 1658.507177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.526245] LNet: 76884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.532816] LNetError: 76884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.537152] LNet: Removed LNI 192.168.201.2@tcp1 [ 1659.567193] LNet: Removed LNI 192.168.201.2@tcp2 [ 1659.599321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.670947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.698071] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1660.410968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.432270] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1661.145760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1661.179462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.2@tcp [ 1661.215790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp1 192.168.201.2@tcp1 [ 1661.248439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp -d 192.168.201.2@tcp -e network_timeout [ 1661.284109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -e network_timeout [ 1661.315991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -e network_timeout [ 1661.347742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -e network_timeout [ 1661.378892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1661.397147] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1661.419480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1661.513683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1661.629467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1661.663116] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1667.461457] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:47:03 (1788587223) [ 1667.548390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.584968] LNet: 77746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.593055] LNetError: 77746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.599793] LNetError: 77746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1667.610940] LNet: Removed LNI 192.168.201.2@tcp [ 1668.646265] LNet: Removed LNI 192.168.201.2@tcp1 [ 1668.697136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.842409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1669.134891] LNet: 77789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.464521] Key type .llcrypt unregistered [ 1669.467376] Key type ._llcrypt unregistered [ 1679.924483] Key type ._llcrypt registered [ 1679.927500] Key type .llcrypt registered [ 1681.232525] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:47:17 (1788587237) [ 1681.663365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1681.680561] alg: No test for adler32 (adler32-zlib) [ 1682.595338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.814549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1685.439896] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:47:21 (1788587241) [ 1685.549985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.580583] LNet: 78877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.625274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.687458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.717374] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1686.412645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1686.437847] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1687.249396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1687.281875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp1 [ 1687.315978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp1 [ 1692.614393] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:47:28 (1788587248) [ 1694.137344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg102-server [ 1695.641866] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:47:31 (1788587251) [ 1697.204808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg102-server [ 1698.716310] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:47:34 (1788587254) [ 1700.458776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg102-server [ 1702.081613] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:47:38 (1788587258) [ 1703.453819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg102-server [ 1705.335636] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:47:41 (1788587261) [ 1706.849656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg102-server [ 1709.045233] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:47:44 (1788587264) [ 1710.969081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg102-server [ 1712.672873] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:47:48 (1788587268) [ 1714.779091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg102-server [ 1716.437500] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:47:52 (1788587272) [ 1718.692485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg102-server [ 1720.543191] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:47:56 (1788587276) [ 1722.216478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg102-server [ 1723.946324] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:47:59 (1788587279) [ 1724.225673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.272662] LNet: 82761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.281773] LNetError: 82761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.288305] LNet: Removed LNI 192.168.201.2@tcp [ 1725.350662] LNet: Removed LNI 192.168.201.2@tcp1 [ 1725.442594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.570800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.631792] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1726.608781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1726.709856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1726.977467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.999607] LNet: 82909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.009554] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.013663] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1728.045767] LNet: Removed LNI 192.168.201.2@tcp [ 1728.110202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.228980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1728.290785] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1729.270238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1729.470182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.492996] LNet: 83054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.498818] LNetError: 83054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.523945] LNet: Removed LNI 192.168.201.2@tcp [ 1729.542243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.631774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.659733] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1730.424183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1730.578354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1730.839303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.949707] LNet: Removed LNI 192.168.201.2@tcp [ 1731.986054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.090809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1732.125805] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1733.218747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1733.549527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.585685] LNet: 83346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.594478] LNet: 83346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1733.616663] LNetError: 83346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.626625] LNetError: 83346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1733.646930] LNet: Removed LNI 192.168.201.2@tcp [ 1733.699710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.794839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.836988] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1734.855352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1734.950635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1735.228203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.310709] LNet: Removed LNI 192.168.201.2@tcp [ 1736.386475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.468771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1736.522869] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1737.429132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1737.703132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.720771] LNetError: 83639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.727344] LNetError: 83639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1737.751471] LNet: Removed LNI 192.168.201.2@tcp [ 1737.806576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.898175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.933547] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1738.871646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1738.957292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1739.160172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.270051] LNet: Removed LNI 192.168.201.2@tcp [ 1740.342403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.441921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1740.489386] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1741.524852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1742.969952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.985794] LNet: 83941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.988609] LNet: 83941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1744.047578] LNet: Removed LNI 192.168.201.2@tcp [ 1744.092392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.165656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.191048] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1744.888672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1744.954329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1746.505452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.534919] LNetError: 84098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.539705] LNetError: 84098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1747.591493] LNet: Removed LNI 192.168.201.2@tcp [ 1747.625280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.726751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1747.758199] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1748.503650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1748.810418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.942718] LNet: Removed LNI 192.168.201.2@tcp [ 1750.010345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.112326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.171954] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1750.825570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1750.926640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1751.188318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.231494] LNet: Removed LNI 192.168.201.2@tcp [ 1752.297869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.403264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1752.438538] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1753.225742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1753.443358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.534461] LNet: Removed LNI 192.168.201.2@tcp [ 1754.554879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.627869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.650296] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1755.393216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1755.480290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.523610] LNet: Removed LNI 192.168.201.2@tcp [ 1755.548521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.641624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.669592] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1756.438147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1756.527533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.565690] LNet: Removed LNI 192.168.201.2@tcp [ 1756.596839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.678788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1756.814690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.859466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.931640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1757.081326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.170791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.241238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1757.392083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.483880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.556333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1757.595675] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1758.402467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.449225] LNet: Removed LNI 192.168.201.2@tcp [ 1758.490692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.582442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1764.881987] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:48:40 (1788587320) [ 1765.078813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.110532] LNet: 85557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.116409] LNet: 85557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1765.135308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.183065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1765.207416] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1766.626828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.646971] LNetError: 85696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.651341] LNetError: 85696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1766.661689] LNet: Removed LNI 192.168.201.2@tcp231 [ 1766.676163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.742723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1766.763337] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1766.768467] LNet: 85704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.272707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1767.311063] LNet: Removed LNI 192.168.201.2@tcp231 [ 1767.340622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1767.370823] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1768.976506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.030371] LNet: Removed LNI 192.168.201.2@tcp231 [ 1769.109737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.193736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1769.220458] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/0] [ 1769.230579] LNet: 85856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.682347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1769.734822] LNet: Removed LNI 192.168.201.2@tcp231 [ 1769.759598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1769.805207] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/0] [ 1770.974024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.023056] LNet: Removed LNI 192.168.201.2@tcp231 [ 1771.047649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.131894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1771.155154] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/60] [ 1771.159520] LNet: 86007:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1771.701506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1771.741760] LNet: Removed LNI 192.168.201.2@tcp231 [ 1771.758291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1771.791428] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/60] [ 1773.036316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.079599] LNet: Removed LNI 192.168.201.2@tcp231 [ 1773.107282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.200419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1773.221512] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1773.226846] LNet: 86159:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1773.697978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1773.750546] LNet: Removed LNI 192.168.201.2@tcp231 [ 1773.765166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1773.816663] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1781.470253] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:48:57 (1788587337) [ 1781.552572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.580959] LNet: Removed LNI 192.168.201.2@tcp231 [ 1781.602339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.673476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1781.702292] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1782.419785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1782.469852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1782.502388] LNet: Removed LNI 192.168.201.2@tcp232 [ 1782.518331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1782.558932] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1783.381600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1783.414376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1783.455850] LNet: Removed LNI 192.168.201.2@tcp232 [ 1783.479960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1783.499300] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1784.102806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1784.137407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1784.162539] LNet: Removed LNI 192.168.201.2@tcp232 [ 1784.177303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1784.198727] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1784.872693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1784.913291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1784.940123] LNet: Removed LNI 192.168.201.2@tcp232 [ 1784.960795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1790.554302] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:49:06 (1788587346) [ 1791.686278] Key type .llcrypt unregistered [ 1791.688696] Key type ._llcrypt unregistered [ 1797.753922] Key type ._llcrypt registered [ 1797.756103] Key type .llcrypt registered [ 1798.004871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1798.029384] alg: No test for adler32 (adler32-zlib) [ 1798.996449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1799.032466] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1807.577709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1817.388345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.446659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1817.475680] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1818.900325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 2 [ 1818.935810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 2 [ 1818.980440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 2 [ 1819.024813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 2 [ 1819.071653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_interrupt -r 2 [ 1819.109167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 2 [ 1819.154358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_interrupt -r 2 [ 1819.209482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 2 [ 1819.271948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1819.302149] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1819.310205] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1819.685264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.750425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.788975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.833805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.859823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1819.890697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.912698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.938649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.965700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.987420] LNet: 88660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1820.007429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1820.037808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1820.072674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1820.125664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1820.191660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1820.244588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.844830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 2 [ 1820.884619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 2 [ 1820.922653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 2 [ 1820.975718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 2 [ 1821.027369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_dropped -r 2 [ 1821.081327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 2 [ 1821.130913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_dropped -r 2 [ 1821.175855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 2 [ 1821.240107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1821.272790] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1821.566472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.610720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1821.649323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.681497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.720332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1821.758396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.797444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.819711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.854314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.872096] LNet: 88782:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.896766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1821.938611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1821.970325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.002262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1822.033957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1822.072571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1822.653210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 2 [ 1822.721347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 2 [ 1822.761300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 2 [ 1822.824301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 2 [ 1822.875362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_aborted -r 2 [ 1822.943439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 2 [ 1823.021768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_aborted -r 2 [ 1823.050241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 2 [ 1823.081224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1823.100161] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1823.319351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1823.345540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1823.368939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1823.392983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1823.421121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1823.444939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1823.475361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1823.501622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1823.524303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1823.537975] LNet: 88905:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1823.554427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1823.577761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1823.604940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1823.633293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1823.661427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1823.691748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1824.472348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 2 [ 1824.505519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 2 [ 1824.549134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 2 [ 1824.586486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 2 [ 1824.636597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_no_route -r 2 [ 1824.666443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 2 [ 1824.703614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_no_route -r 2 [ 1824.768492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 2 [ 1824.834680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1825.128805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1825.167211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1825.216490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1825.250179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1825.310812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1825.365512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1825.404518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1825.464513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1825.521776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1825.553629] LNet: 89027:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1825.605425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1825.648667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1825.699514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1825.745430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1825.777400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1825.800654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1826.352424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 2 [ 1826.394264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 2 [ 1826.428728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 2 [ 1826.461379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 2 [ 1826.521418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_timeout -r 2 [ 1826.554920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 2 [ 1826.596745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_timeout -r 2 [ 1826.645708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 2 [ 1826.708397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1826.744886] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1826.755841] LNet: Skipped 1 previous similar message [ 1827.053638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1827.081882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1827.116540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.145532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.181884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1827.238333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.302438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.360985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.400447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.447175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1827.491448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1827.528822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.566512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1827.611091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1827.658911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1828.403505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 2 [ 1828.439393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 2 [ 1828.473450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 2 [ 1828.521792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 2 [ 1828.572556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_dropped -r 2 [ 1828.650283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 2 [ 1828.721262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_dropped -r 2 [ 1828.760320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 2 [ 1828.793328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1828.813361] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 1829.000138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1829.033530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1829.061973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1829.102474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1829.169527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1829.246546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.310587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.359571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.408434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.465187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.507602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.532815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.564963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.596567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.643160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1835.075992] LNet: 89346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.081450] LNetError: 89346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.087706] LNet: Removed LNI 192.168.201.2@tcp [ 1835.104967] LNet: Removed LNI 192.168.201.2@tcp1 [ 1835.503051] Key type .llcrypt unregistered [ 1835.505746] Key type ._llcrypt unregistered [ 1854.835754] Key type ._llcrypt registered [ 1854.839847] Key type .llcrypt registered [ 1856.168738] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:50:11 (1788587411) [ 1856.861178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1856.891923] alg: No test for adler32 (adler32-zlib) [ 1857.905604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.951794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1857.996563] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1859.229664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1859.275288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1859.300132] LNetError: 90323:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1859.418986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.2@tcp [ 1859.615184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1868.694825] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:50:23 (1788587423) [ 1868.958128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.001328] LNet: 90909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.024461] LNetError: 90909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.050677] LNet: Removed LNI 192.168.201.2@tcp [ 1869.113556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.342799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1869.423278] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1871.067136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1871.088689] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1872.968540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.2@tcp [ 1873.018247] LNetError: 91181:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1873.311760] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1873.318891] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1874.555809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.2@tcp [ 1883.952377] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:50:39 (1788587439) [ 1886.089853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg102-server [ 1888.069219] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:50:43 (1788587443) [ 1890.265878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg102-server [ 1892.318064] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:50:47 (1788587447) [ 1894.395806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg102-server [ 1896.128892] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:50:52 (1788587452) [ 1896.250632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.295187] LNet: 92799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.307030] LNetError: 92799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.314723] LNet: Removed LNI 192.168.201.2@tcp [ 1896.337623] LNet: Removed LNI 192.168.201.2@tcp1 [ 1896.360961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.408278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.427455] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1896.992989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1897.056093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1897.092370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1897.194445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1897.227840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1897.297227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1903.965068] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:50:59 (1788587459) [ 1904.094409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.137196] LNet: 93530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.152777] LNetError: 93530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.155568] LNetError: 93530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1904.175476] LNet: Removed LNI 192.168.201.2@tcp [ 1904.195797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.293248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.312220] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1911.226788] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:51:07 (1788587467) [ 1911.350659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.376506] LNet: 94237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.390498] LNetError: 94237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.406647] LNet: Removed LNI 192.168.201.2@tcp [ 1911.450046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.501717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.516684] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1919.084480] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:51:14 (1788587474) [ 1920.725872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1922.549710] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:51:18 (1788587478) [ 1922.965618] LNet: 95277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.970749] LNetError: 95277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.979754] LNet: Removed LNI 192.168.201.2@tcp [ 1923.878624] Key type .llcrypt unregistered [ 1923.880862] Key type ._llcrypt unregistered [ 1931.908533] Key type ._llcrypt registered [ 1931.910929] Key type .llcrypt registered [ 1932.094295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.142604] alg: No test for adler32 (adler32-zlib) [ 1933.087459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1933.108561] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1941.999681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1950.105984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.102@tcp [ 1950.922927] LNet: 95694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.941715] LNetError: 95694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.039970] LNet: Removed LNI 192.168.201.2@tcp [ 1952.067859] Key type .llcrypt unregistered [ 1952.069258] Key type ._llcrypt unregistered [ 1967.318610] Key type ._llcrypt registered [ 1967.321923] Key type .llcrypt registered [ 1969.405886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1971.218612] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1973.547888] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:52:08 (1788587528) [ 1976.003610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1978.562050] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:52:13 (1788587533) [ 1981.343438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg102-server [ 1983.980962] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:52:19 (1788587539) [ 1986.534455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg102-server [ 1988.882641] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:52:24 (1788587544) [ 1990.400807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.475208] alg: No test for adler32 (adler32-zlib) [ 1991.449219] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1991.585302] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1993.287148] Key type lgssc registered [ 1994.857132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.111801] Key type lgssc unregistered [ 2016.470766] LNet: 98867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.489449] LNetError: 98867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.552238] LNet: Removed LNI 192.168.201.2@tcp [ 2017.707428] Key type .llcrypt unregistered [ 2017.709833] Key type ._llcrypt unregistered [ 2034.591575] Key type ._llcrypt registered [ 2034.593529] Key type .llcrypt registered [ 2035.775774] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:53:11 (1788587591) [ 2036.091017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2036.107605] alg: No test for adler32 (adler32-zlib) [ 2037.040067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.123949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2037.178758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2037.224285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2037.268862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2043.961603] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:53:19 (1788587599) [ 2044.200587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.232208] LNet: 100280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.256257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.311331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2044.332641] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2045.184368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.202866] LNet: 100424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.207559] LNetError: 100424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.220640] LNet: Removed LNI 192.168.201.2@tcp [ 2045.232825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.269729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2045.289236] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2045.800612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.828402] LNetError: 100452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.851813] LNet: Removed LNI 192.168.201.2@tcp [ 2045.871507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.937729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2045.987333] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2052.880563] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:53:28 (1788587608) [ 2053.190793] LNet: 101042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.203519] LNet: 101042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2053.218664] LNetError: 101042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.239714] LNet: Removed LNI 192.168.201.2@tcp [ 2053.870711] Key type .llcrypt unregistered [ 2053.873114] Key type ._llcrypt unregistered [ 2060.331138] Key type ._llcrypt registered [ 2060.333159] Key type .llcrypt registered [ 2060.464613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2060.481317] alg: No test for adler32 (adler32-zlib) [ 2061.498981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.619920] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2061.631857] LNet: *** cfs_fail_loc=e003, val=100*** [ 2062.540239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2062.565391] LNet: *** cfs_fail_loc=e003, val=100*** [ 2062.572276] LNet: Skipped 2 previous similar messages [ 2062.585793] Ping reply received with mapping: [ 2062.591619] found metadata reply NI[192.168.201.2@tcp] [ 2062.595506] 02 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2062.603767] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2062.610213] 5a 5a 5a 5a 5a 5a 5a 5a [ 2062.790459] LNet: 101437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.797274] LNetError: 101437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.826706] LNet: Removed LNI 192.168.201.2@tcp [ 2063.525532] Key type .llcrypt unregistered [ 2063.527402] Key type ._llcrypt unregistered [ 2074.959590] Key type ._llcrypt registered [ 2074.962197] Key type .llcrypt registered [ 2076.004329] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:53:52 (1788587632) [ 2076.664484] Key type .llcrypt unregistered [ 2076.670466] Key type ._llcrypt unregistered [ 2082.949052] Key type ._llcrypt registered [ 2082.950808] Key type .llcrypt registered [ 2083.256913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2083.294221] alg: No test for adler32 (adler32-zlib) [ 2084.491789] LNetError: 102472:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2085.016637] Key type .llcrypt unregistered [ 2085.020254] Key type ._llcrypt unregistered [ 2090.352981] Key type ._llcrypt registered [ 2090.355176] Key type .llcrypt registered [ 2092.025661] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:54:07 (1788587647) [ 2092.845491] Key type .llcrypt unregistered [ 2092.849298] Key type ._llcrypt unregistered [ 2099.494940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2099.507099] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2102.015308] Key type ._llcrypt registered [ 2102.017359] Key type .llcrypt registered [ 2102.205831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.216725] alg: No test for adler32 (adler32-zlib) [ 2103.142484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.201044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2112.683958] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:54:28 (1788587668) [ 2112.845804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.889318] LNet: 104239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.903671] LNetError: 104239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2112.938176] LNet: Removed LNI 10.1.2.3@tcp [ 2113.049110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.129735] Key type .llcrypt unregistered [ 2114.138804] Key type ._llcrypt unregistered [ 2121.999152] Key type ._llcrypt registered [ 2122.001825] Key type .llcrypt registered [ 2122.272525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2122.318923] alg: No test for adler32 (adler32-zlib) [ 2123.279077] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2123.379717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2123.581952] LNet: 104530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.589063] LNetError: 104530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.610296] LNet: Removed LNI 192.168.201.2@tcp [ 2124.234122] Key type .llcrypt unregistered [ 2124.238978] Key type ._llcrypt unregistered [ 2135.815764] Key type ._llcrypt registered [ 2135.819562] Key type .llcrypt registered [ 2137.272825] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:54:52 (1788587692) [ 2138.250389] Key type .llcrypt unregistered [ 2138.256471] Key type ._llcrypt unregistered [ 2145.280786] Key type ._llcrypt registered [ 2145.284696] Key type .llcrypt registered [ 2145.539923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2145.554090] alg: No test for adler32 (adler32-zlib) [ 2153.165816] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:55:09 (1788587709) [ 2153.428043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.469628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.517629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.539346] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2154.426549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.521738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2154.572063] LNet: Removed LNI 192.168.201.2@tcp [ 2154.884322] LNet: 106269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.328535] Key type .llcrypt unregistered [ 2155.330975] Key type ._llcrypt unregistered [ 2169.019551] Key type ._llcrypt registered [ 2169.020711] Key type .llcrypt registered [ 2170.541836] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:55:26 (1788587726) [ 2170.841980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.861330] alg: No test for adler32 (adler32-zlib) [ 2171.829754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.884162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.901497] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2172.745570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2172.808761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.2@tcp [ 2172.828169] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2172.832217] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2172.835635] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2172, 0, 0 [ 2172.840680] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2172.845570] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2172.853890] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2172.859129] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2172, 0, 0 [ 2172.864105] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2172.866679] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2172.873105] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2172.875905] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2172, 0, 2222 [ 2172.881374] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2172.883959] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2172.886543] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2172.889502] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2172, 0, 2222 [ 2172.897797] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2179.373619] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:55:35 (1788587735) [ 2179.923752] LNet: 107806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.929987] LNetError: 107806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.966850] LNet: Removed LNI 192.168.201.2@tcp [ 2181.820451] Key type .llcrypt unregistered [ 2181.825502] Key type ._llcrypt unregistered [ 2189.570298] Key type ._llcrypt registered [ 2189.576176] Key type .llcrypt registered [ 2189.808877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2189.829766] alg: No test for adler32 (adler32-zlib) [ 2190.774532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2190.818686] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2198.826237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2208.997681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.083992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2209.129673] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 2210.232390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2210.398394] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.457346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2210.469419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2212.840506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2212.862312] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2223.839730] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:56:19 (1788587779) [ 2224.141985] LNet: 109310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.151636] LNetError: 109310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.183757] LNet: Removed LNI 192.168.201.2@tcp [ 2225.188478] LNetError: 109310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2225.205409] LNet: Removed LNI 192.168.201.2@tcp1 [ 2225.240306] LNet: Removed LNI 10.1.2.3@tcp99 [ 2225.844803] Key type .llcrypt unregistered [ 2225.846711] Key type ._llcrypt unregistered [ 2233.085422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2233.093738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2235.961559] Key type ._llcrypt registered [ 2235.964074] Key type .llcrypt registered [ 2236.423960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2236.469247] alg: No test for adler32 (adler32-zlib) [ 2237.443341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.535510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.579520] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2238.352892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2238.379938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2240.708792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2240.779685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.848142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2240.874436] LNetError: 110051:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2240.918861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2240.968470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2241.622722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2241.757667] LNet: 110071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2241.761505] LNetError: 110071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2241.765494] LNet: Removed LNI 10.1.2.3@tcp [ 2242.792594] LNet: Removed LNI 192.168.201.2@tcp [ 2243.859483] Key type .llcrypt unregistered [ 2243.863712] Key type ._llcrypt unregistered [ 2259.215956] Key type ._llcrypt registered [ 2259.218869] Key type .llcrypt registered [ 2260.832987] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:56:56 (1788587816) [ 2261.258505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.276434] alg: No test for adler32 (adler32-zlib) [ 2262.255051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.320975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.356554] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2263.095765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2263.174677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg102-client@tcp [ 2270.460798] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:57:06 (1788587826) [ 2270.566774] LNet: 111594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.576548] LNetError: 111594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.608221] LNet: Removed LNI 192.168.201.2@tcp [ 2271.337570] Key type .llcrypt unregistered [ 2271.342789] Key type ._llcrypt unregistered [ 2278.978140] Key type ._llcrypt registered [ 2278.987728] Key type .llcrypt registered [ 2279.177372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2279.206679] alg: No test for adler32 (adler32-zlib) [ 2280.192792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2280.222534] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2280.248786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2287.835485] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:57:23 (1788587843) [ 2289.199769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2290.852898] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:57:26 (1788587846) [ 2292.463300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2294.386604] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:57:30 (1788587850) [ 2294.534176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.579379] LNet: 113016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2294.588110] LNetError: 113016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2294.604501] LNet: Removed LNI 192.168.201.2@tcp [ 2294.650058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.712785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2294.737477] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2295.217894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2295.254837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2301.895814] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:57:37 (1788587857) [ 2301.994221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.038774] LNet: 113725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.045543] LNetError: 113725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.065148] LNet: Removed LNI 192.168.201.2@tcp [ 2302.133842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.171214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2302.199380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2302.848256] Key type .llcrypt unregistered [ 2302.849742] Key type ._llcrypt unregistered [ 2313.387154] Key type ._llcrypt registered [ 2313.389916] Key type .llcrypt registered [ 2314.782959] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:57:50 (1788587870) [ 2315.134577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.149164] alg: No test for adler32 (adler32-zlib) [ 2316.128127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.260485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.292992] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2317.089432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2317.157361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2324.033326] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:58:00 (1788587880) [ 2324.124424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.174995] LNet: 115278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.180931] LNetError: 115278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.222318] LNet: Removed LNI 192.168.201.2@tcp [ 2325.307927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.387823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2325.424755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2331.476548] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:58:07 (1788587887) [ 2331.672374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.719813] LNet: 115861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.741001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.875908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2332.495079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2332.531359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2332.644803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2333.180376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.233573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.349698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.378229] LNet: 115897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.438759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.522418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2334.094219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.158590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2334.248539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2334.815648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.850344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2341.499185] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:58:17 (1788587897) [ 2342.070909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.098796] LNet: 116499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.178891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.264737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.339184] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2343.424573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2343.441935] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2344.543625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2344.655594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2344.918764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.969377] LNetError: 116778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.981289] LNet: Removed LNI 192.168.201.2@tcp [ 2344.990874] LNet: Removed LNI 192.168.201.2@tcp2 [ 2345.032711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.144911] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2345.151454] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2345.159367] LNet: 116784:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2345.971996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2346.314745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.346552] LNet: 116821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.353368] LNet: 116821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2346.365747] LNetError: 116821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.373865] LNetError: 116821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2346.387364] LNet: Removed LNI 192.168.201.2@tcp [ 2346.404316] LNet: Removed LNI 192.168.201.2@tcp2 [ 2346.432979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.530453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2346.575149] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2347.612466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2347.653432] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2348.676935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2348.744601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2348.846847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2349.159475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.203074] LNetError: 117106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.210894] LNetError: 117106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2349.217707] LNet: Removed LNI 192.168.201.2@tcp [ 2350.248895] LNet: Removed LNI 192.168.201.2@tcp2 [ 2350.293799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.444076] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2350.460888] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2350.479197] LNet: 117112:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.437289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2351.837966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.880730] LNet: Removed LNI 192.168.201.2@tcp [ 2352.937308] LNet: Removed LNI 192.168.201.2@tcp2 [ 2352.989389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.087146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.153235] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2354.256543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2354.275590] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2355.443741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2355.519520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2355.745905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.770192] LNet: 117429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.780358] LNet: 117429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2355.789334] LNetError: 117429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.800925] LNetError: 117429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2355.810450] LNet: Removed LNI 192.168.201.2@tcp [ 2355.824299] LNet: Removed LNI 192.168.201.2@tcp2 [ 2355.865552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.934246] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2355.939952] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2355.946525] LNet: 117435:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.784442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2357.066381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.111149] LNet: Removed LNI 192.168.201.2@tcp [ 2358.121198] LNet: Removed LNI 192.168.201.2@tcp2 [ 2358.156459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.222305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2358.255610] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2359.045692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2359.081950] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2360.104858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2360.165449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2360.264236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2360.545965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.631379] LNet: Removed LNI 192.168.201.2@tcp [ 2362.662037] LNet: Removed LNI 192.168.201.2@tcp2 [ 2362.695859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.836234] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2362.852833] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2362.866437] LNet: 117763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.791403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2364.127883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.149753] LNetError: 117800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.159785] LNetError: 117800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2364.170283] LNet: Removed LNI 192.168.201.2@tcp [ 2365.224570] LNet: Removed LNI 192.168.201.2@tcp2 [ 2365.294424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.408256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.443830] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2366.428100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2366.465989] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2367.681767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2367.709750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2367.814664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2368.196269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.247932] LNet: Removed LNI 192.168.201.2@tcp [ 2369.266689] LNet: Removed LNI 192.168.201.2@tcp2 [ 2369.303716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.427792] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2369.439157] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2369.452263] LNet: 118103:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2370.129565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2370.539753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.615438] LNet: Removed LNI 192.168.201.2@tcp [ 2371.631651] LNet: Removed LNI 192.168.201.2@tcp2 [ 2371.708874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2371.806888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2371.830030] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2372.525345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2372.572550] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2373.612644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2373.656706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2373.700910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2373.764453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2374.073192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.094671] LNet: 118456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.098810] LNet: 118456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2375.136972] LNet: Removed LNI 192.168.201.2@tcp [ 2376.229859] LNet: Removed LNI 192.168.201.2@tcp2 [ 2376.252851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.297409] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2376.300881] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2376.708391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2384.815403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2386.690837] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:59:02 (1788587942) [ 2386.992339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.020760] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.025310] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2387.031180] LNet: Removed LNI 192.168.201.2@tcp [ 2388.070831] LNet: Removed LNI 192.168.201.2@tcp2 [ 2388.101179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.390764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2388.421815] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2389.062931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2389.082860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2390.214271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2390.236331] LNet: Removed LNI 10.1.2.3@tcp [ 2390.253714] LNet: Removed LNI 192.168.201.2@tcp [ 2390.304370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.392771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2390.436202] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2390.450658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2391.007848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2391.045451] LNet: Removed LNI 192.168.201.2@tcp [ 2391.066392] LNet: Removed LNI 10.1.2.3@tcp [ 2398.487768] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:59:13 (1788587953) [ 2398.570877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.638554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.732254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2398.791476] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2399.963060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2400.044570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2400.129976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.204526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.251758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.327384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.393580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.461799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.536186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.608710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.711491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2400.780460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2401.050572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2408.545877] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 01:59:24 (1788587964) [ 2408.686977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.745899] LNet: 120832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.751189] LNet: 120832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2409.858969] LNet: Removed LNI 192.168.201.2@tcp [ 2409.885424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.962215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2410.005542] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2410.683862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2410.708621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2410.741408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.770430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.804087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.830529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.862516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.884322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.913934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.943665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2410.982993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.014445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.088602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2411.129599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.202301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.266352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.349152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.408329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.453627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.507638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.559592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.707682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2411.809585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2419.124815] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:59:35 (1788587975) [ 2419.469617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2419.509417] LNetError: 121593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.519919] LNetError: 121593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2419.548402] LNet: Removed LNI 192.168.201.2@tcp [ 2419.567959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.695723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2419.875417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2419.904223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2419.912495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2422.293211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2422.318279] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2422.887789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2422.904892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2425.526732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2425.613928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2425.689411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.2@tcp -m PUT -e local_timeout -r 1 [ 2425.743605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2425.797768] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2425.806278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2426.090198] LNetError: 121661:0:(lib-move.c:4794:lnet_parse()) 192.168.201.2@tcp, src 192.168.201.2@tcp: Dropping PUT (error -108 looking up sender) [ 2427.115716] LNet: Removed LNI 10.1.2.3@tcp [ 2428.212845] LNet: Removed LNI 192.168.201.2@tcp [ 2429.027290] Key type .llcrypt unregistered [ 2429.046797] Key type ._llcrypt unregistered [ 2444.737297] Key type ._llcrypt registered [ 2444.738992] Key type .llcrypt registered [ 2446.871372] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:00:01 (1788588001) [ 2447.640608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.661309] alg: No test for adler32 (adler32-zlib) [ 2448.614833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2449.138878] LNet: 122969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.102932] Key type .llcrypt unregistered [ 2450.106926] Key type ._llcrypt unregistered [ 2458.080568] Key type ._llcrypt registered [ 2458.083617] Key type .llcrypt registered [ 2458.370314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2458.414709] alg: No test for adler32 (adler32-zlib) [ 2459.400424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2459.443815] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2466.314771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2466.490690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2466.956720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.102@tcp [ 2467.092784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2467.114751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2467.129887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2467.143380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2467.163474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2467.179849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2467.193607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2467.212857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2467.235300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2467.247536] LNet: 123418:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2467.264439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2467.295447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2467.317827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2467.341349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2467.373992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2467.393097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2468.078636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2468.305324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2468.345193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2468.401448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2468.439627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2468.489364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2468.529084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2468.562181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2468.597932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2468.643112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2468.671152] LNet: 123537:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2468.702835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2468.704856] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 2468.752899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2468.782256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2468.816524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2468.854529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2468.874703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2469.546409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.102@tcp [ 2469.825724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2469.854507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2469.886220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2469.930587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2469.963177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2469.998961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2470.027864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2470.056940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2470.091176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2470.108467] LNet: 123656:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2470.127832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2470.159401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2470.188410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2470.214646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2470.241659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2470.270948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2470.959490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2471.309548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2471.360041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2471.399277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2471.420477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2471.463503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2471.498818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2471.532440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2471.575337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2471.611526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2471.663874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2471.697755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2471.720885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2471.747293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2471.779396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2471.815254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2472.079131] LNet: 123821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.087119] LNetError: 123821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.108207] LNet: Removed LNI 192.168.201.2@tcp [ 2472.633937] Key type .llcrypt unregistered [ 2472.642143] Key type ._llcrypt unregistered [ 2489.404512] Key type ._llcrypt registered [ 2489.406453] Key type .llcrypt registered [ 2490.963799] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:00:46 (1788588046) [ 2491.402321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.439523] alg: No test for adler32 (adler32-zlib) [ 2492.394051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.593157] LNet: 124706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2492.969458] Key type .llcrypt unregistered [ 2492.972082] Key type ._llcrypt unregistered [ 2500.487889] Key type ._llcrypt registered [ 2500.488929] Key type .llcrypt registered [ 2500.694615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2500.709604] alg: No test for adler32 (adler32-zlib) [ 2501.710796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2501.744661] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2511.959771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2522.060768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.110431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2522.152695] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 2523.173599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2523.959593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.102@tcp [ 2524.316405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2524.334960] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 2524.376034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2524.423518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2524.468317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2524.508471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2524.546636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2524.581171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2524.653644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2524.706444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2524.761555] LNet: 125371:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2524.799438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2524.874796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2524.948570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2525.000550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2525.055882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2525.092885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2526.004995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2526.310803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2526.359911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2526.390325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2526.436496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2526.472342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2526.526418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2526.574770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2526.630523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2526.686239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2526.715064] LNet: 125490:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2526.734092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2526.768505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2526.797163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2526.830281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2526.860587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2526.902322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2527.457465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.102@tcp [ 2527.662696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2527.684506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2527.706188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2527.731484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2527.756635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2527.777425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2527.806381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2527.827137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2527.852825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2527.864898] LNet: 125609:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2527.886549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2527.928502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2527.957650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2527.998466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2528.040327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2528.091099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2528.721407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2528.959996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2529.008765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2529.031460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2529.094391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2529.137554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2529.195393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2529.243076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2529.281608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2529.329245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2529.387796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2529.448215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2529.510970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2529.565356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2529.594965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2529.631624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2534.638898] LNet: 125810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.644911] LNetError: 125810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.656102] LNet: Removed LNI 192.168.201.2@tcp [ 2534.681245] LNet: Removed LNI 192.168.201.2@tcp1 [ 2535.180886] Key type .llcrypt unregistered [ 2535.183522] Key type ._llcrypt unregistered [ 2552.150405] Key type ._llcrypt registered [ 2552.151727] Key type .llcrypt registered [ 2553.396210] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 02:01:49 (1788588109) [ 2553.817515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2553.835797] alg: No test for adler32 (adler32-zlib) [ 2554.788512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2554.867986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2554.959440] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2555.973790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2555.993920] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2556.794821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2556.895605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2557.919211] LNet: Removed LNI 192.168.201.2@tcp [ 2568.159712] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 02:02:02 (1788588122) [ 2570.069432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2572.350654] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:02:07 (1788588127) [ 2580.737223] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:02:16 (1788588136) [ 2588.930519] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:02:24 (1788588144) [ 2598.311358] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:02:34 (1788588154) [ 2607.019572] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:02:42 (1788588162) [ 2615.877300] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:02:50 (1788588170) [ 2623.541648] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:02:59 (1788588179) [ 2634.686781] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:03:09 (1788588189) [ 2644.840919] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:03:20 (1788588200) [ 2652.933775] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:03:28 (1788588208) [ 2659.401136] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:03:35 (1788588215) [ 2666.807187] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:03:42 (1788588222) [ 2674.587634] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:03:50 (1788588230) [ 2682.095416] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:03:58 (1788588238) [ 2691.079926] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:04:06 (1788588246) [ 2699.357572] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:04:15 (1788588255) [ 2707.560466] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:04:23 (1788588263) [ 2714.976884] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:04:30 (1788588270) [ 2715.211757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2715.261153] LNet: 137496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2715.284102] LNetError: 137496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2715.324009] LNet: Removed LNI 192.168.201.2@tcp2 [ 2715.358566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2715.471627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2715.495651] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2716.666303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2716.701684] LNet: 137644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2716.710865] LNetError: 137644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2716.727795] LNet: Removed LNI 192.168.201.2@tcp [ 2716.753836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2716.822856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2716.851137] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2724.976697] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:04:40 (1788588280) [ 2733.125863] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:04:49 (1788588289) [ 2743.324781] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:04:58 (1788588298) [ 2751.423834] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:05:07 (1788588307) [ 2760.051569] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:05:15 (1788588315) [ 2767.062983] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:05:22 (1788588322) [ 2775.275659] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:05:30 (1788588330) [ 2784.765963] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:05:40 (1788588340) [ 2794.023467] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:05:49 (1788588349) [ 2801.768807] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:05:57 (1788588357) [ 2810.307250] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:06:05 (1788588365) [ 2818.196444] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:06:13 (1788588373) [ 2826.520588] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2606 sec ========== 02:06:22 (1788588382) [ 2826.923308] LNet: 145113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2826.927415] LNetError: 145113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2826.946411] LNet: Removed LNI 192.168.201.2@tcp [ 2827.700468] Key type .llcrypt unregistered [ 2827.704327] Key type ._llcrypt unregistered [ 2840.839586] Key type ._llcrypt registered [ 2840.851056] Key type .llcrypt registered [ 2841.444727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2841.453767] alg: No test for adler32 (adler32-zlib) [ 2842.674918] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2842.913238] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2844.647525] Key type lgssc registered [ 2846.370578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2916.402826] Lustre: Mounted lustre-client - version 2.17.58_3_g357faf0 [ 2923.180765] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.691690] Lustre: Unmounted lustre-client [ 2963.046361] Key type lgssc unregistered [ 2963.260387] LNet: 148065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2963.266654] LNetError: 148065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2963.286943] LNet: Removed LNI 192.168.201.2@tcp [ 2964.123313] Key type .llcrypt unregistered [ 2964.126295] Key type ._llcrypt unregistered