[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 430739665 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006817] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010057] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.013037] *** VALIDATE selinux *** [ 0.021140] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025786] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028109] *** VALIDATE tmpfs *** [ 0.030253] *** VALIDATE proc *** [ 0.031229] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.034081] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040828] debug: unmapping init [mem 0xffffffff9f059000-0xffffffff9f060fff] [ 0.042210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049016] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051311] rcu: Hierarchical SRCU implementation. [ 0.053551] smp: Bringing up secondary CPUs ... [ 0.054619] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.058215] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150489] node 0 deferred pages initialised in 87ms [ 0.154125] devtmpfs: initialized [ 0.156301] x86/mm: Memory block size: 128MB [ 0.159970] gcov: version magic: 0x41383552 [ 0.162298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.165099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167276] pinctrl core: initialized pinctrl subsystem [ 0.169113] [ 0.169767] ************************************************************* [ 0.171011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174014] ** ** [ 0.176012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178011] ** ** [ 0.180012] ** This means that this kernel is built to expose internal ** [ 0.182014] ** IOMMU data structures, which may compromise security on ** [ 0.183009] ** your system. ** [ 0.186010] ** ** [ 0.187008] ** If you see this message and you are not debugging the ** [ 0.188009] ** kernel, report this immediately to your vendor! ** [ 0.190009] ** ** [ 0.192009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193008] ************************************************************* [ 0.195590] NET: Registered protocol family 16 [ 0.196344] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.198045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203515] cpuidle: using governor menu [ 0.204603] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206455] PCI: Using configuration type 1 for base access [ 0.208115] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.216162] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.218028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222073] cryptd: max_cpu_qlen set to 1000 [ 0.225247] ACPI: Added _OSI(Module Device) [ 0.227015] ACPI: Added _OSI(Processor Device) [ 0.229034] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.230012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.233046] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237422] ACPI: Interpreter enabled [ 0.239044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239949] ACPI: Using IOAPIC for interrupt routing [ 0.241107] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242266] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.248795] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.257086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.262561] acpiphp: Slot [2] registered [ 0.264139] acpiphp: Slot [5] registered [ 0.265078] acpiphp: Slot [6] registered [ 0.266060] acpiphp: Slot [3] registered [ 0.267088] acpiphp: Slot [4] registered [ 0.269154] acpiphp: Slot [7] registered [ 0.270096] acpiphp: Slot [8] registered [ 0.272117] acpiphp: Slot [9] registered [ 0.273124] acpiphp: Slot [10] registered [ 0.274131] acpiphp: Slot [11] registered [ 0.276108] acpiphp: Slot [12] registered [ 0.277063] acpiphp: Slot [13] registered [ 0.277990] acpiphp: Slot [14] registered [ 0.279054] acpiphp: Slot [15] registered [ 0.280038] acpiphp: Slot [16] registered [ 0.281096] acpiphp: Slot [17] registered [ 0.283075] acpiphp: Slot [18] registered [ 0.283971] acpiphp: Slot [19] registered [ 0.284054] acpiphp: Slot [20] registered [ 0.286073] acpiphp: Slot [21] registered [ 0.287089] acpiphp: Slot [22] registered [ 0.288115] acpiphp: Slot [23] registered [ 0.290099] acpiphp: Slot [24] registered [ 0.290962] acpiphp: Slot [25] registered [ 0.291048] acpiphp: Slot [26] registered [ 0.291833] acpiphp: Slot [27] registered [ 0.293064] acpiphp: Slot [28] registered [ 0.294097] acpiphp: Slot [29] registered [ 0.296096] acpiphp: Slot [30] registered [ 0.297085] acpiphp: Slot [31] registered [ 0.298046] PCI host bridge to bus 0000:00 [ 0.298859] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.302017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.304018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.309205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313047] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316321] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323616] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.328059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.334019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336478] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.339572] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.349016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.360018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.364017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370555] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.381015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.386044] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.407186] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.412019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.416014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.434329] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.436289] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.438384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.441365] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.443189] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.448035] iommu: Default domain type: Passthrough [ 0.450388] SCSI subsystem initialized [ 0.453132] ACPI: bus type USB registered [ 0.455113] usbcore: registered new interface driver usbfs [ 0.457082] usbcore: registered new interface driver hub [ 0.458077] usbcore: registered new device driver usb [ 0.460155] pps_core: LinuxPPS API ver. 1 registered [ 0.461009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.464079] PTP clock support registered [ 0.467052] EDAC MC: Ver: 3.0.0 [ 0.468354] PCI: Using ACPI for IRQ routing [ 0.470892] NetLabel: Initializing [ 0.473019] NetLabel: domain hash size = 128 [ 0.475009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.476078] NetLabel: unlabeled traffic allowed by default [ 0.478112] vgaarb: loaded [ 0.479298] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.481013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.487154] clocksource: Switched to clocksource kvm-clock [ 0.590487] VFS: Disk quotas dquot_6.6.0 [ 0.592174] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.594529] *** VALIDATE ramfs *** [ 0.595768] *** VALIDATE hugetlbfs *** [ 0.597215] pnp: PnP ACPI init [ 0.599098] pnp: PnP ACPI: found 6 devices [ 0.614716] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.618029] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.620018] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.621881] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.624273] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.626721] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.630063] NET: Registered protocol family 2 [ 0.632476] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.637322] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.640852] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.645932] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.649283] TCP: Hash tables configured (established 65536 bind 65536) [ 0.652121] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.655078] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657785] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.660338] NET: Registered protocol family 1 [ 0.663243] RPC: Registered named UNIX socket transport module. [ 0.664600] RPC: Registered udp transport module. [ 0.665697] RPC: Registered tcp transport module. [ 0.666721] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.668287] NET: Registered protocol family 44 [ 0.669382] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.670669] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.672025] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.673580] PCI: CLS 0 bytes, default 64 [ 0.674723] Unpacking initramfs... [ 1.934709] debug: unmapping init [mem 0xffff93697cc64000-0xffff93697ffcffff] [ 1.938728] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.940586] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.943204] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.420625] Initialise system trusted keyrings [ 2.421935] Key type blacklist registered [ 2.423148] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.432265] zbud: loaded [ 2.435229] *** VALIDATE nfs *** [ 2.436547] *** VALIDATE nfs4 *** [ 2.438168] pstore: using deflate compression [ 2.442085] Platform Keyring initialized [ 2.534446] NET: Registered protocol family 38 [ 2.536247] Key type asymmetric registered [ 2.537713] Asymmetric key parser 'x509' registered [ 2.539371] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.541993] io scheduler mq-deadline registered [ 2.543713] io scheduler kyber registered [ 2.545381] io scheduler bfq registered [ 2.547072] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.549691] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.551335] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.553453] ACPI: Power Button [PWRF] [ 2.557472] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.563375] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.573908] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.600587] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.629770] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.634836] Non-volatile memory driver v1.3 [ 2.636098] Linux agpgart interface v0.103 [ 2.662420] virtio_blk virtio1: [vda] 150024 512-byte logical blocks (76.8 MB/73.3 MiB) [ 2.664479] vda: detected capacity change from 0 to 76812288 [ 2.678689] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.680915] vdb: detected capacity change from 0 to 1073741824 [ 2.687231] libphy: Fixed MDIO Bus: probed [ 2.704595] usbcore: registered new interface driver usbserial_generic [ 2.706872] usbserial: USB Serial support registered for generic [ 2.708780] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.711685] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.712890] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.715065] mousedev: PS/2 mouse device common for all mice [ 2.717781] rtc_cmos 00:05: RTC can wake from S4 [ 2.721142] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.722093] rtc_cmos 00:05: registered as rtc0 [ 2.726693] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.730027] intel_pstate: CPU model not supported [ 2.732422] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.733688] hid: raw HID events driver (C) Jiri Kosina [ 2.736615] usbcore: registered new interface driver usbhid [ 2.738583] usbhid: USB HID core driver [ 2.740137] drop_monitor: Initializing network drop monitor service [ 2.740200] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.741629] Initializing XFRM netlink socket [ 2.742081] NET: Registered protocol family 10 [ 2.748771] Segment Routing with IPv6 [ 2.749966] NET: Registered protocol family 17 [ 2.751568] mpls_gso: MPLS GSO support [ 2.757850] RAS: Correctable Errors collector initialized. [ 2.759447] AVX version of gcm_enc/dec engaged. [ 2.760734] AES CTR mode by8 optimization enabled [ 2.837439] sched_clock: Marking stable (2837411100, 0)->(3690081129, -852670029) [ 2.840351] registered taskstats version 1 [ 2.842606] Loading compiled-in X.509 certificates [ 2.845126] zswap: loaded using pool lzo/zbud [ 2.869469] Key type big_key registered [ 2.879626] Key type encrypted registered [ 2.881222] ima: No TPM chip found, activating TPM-bypass! [ 2.883295] ima: Allocated hash algorithm: sha1 [ 2.884290] ima: No architecture policies found [ 2.885376] evm: Initialising EVM extended attributes: [ 2.886646] evm: security.selinux [ 2.887868] evm: security.ima [ 2.888543] evm: security.capability [ 2.890036] evm: HMAC attrs: 0x1 [ 2.892196] rtc_cmos 00:05: setting system clock to 2026-09-14 23:14:26 UTC (1789427666) [ 2.897807] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 2.900407] debug: unmapping init [mem 0xffffffff9ed82000-0xffffffff9f058fff] [ 2.909105] Write protecting the kernel read-only data: 28672k [ 2.912965] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 2.915807] debug: unmapping init [mem 0xffffffff9dd14000-0xffffffff9ddfffff] [ 2.945947] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.951444] systemd[1]: Detected virtualization kvm. [ 2.952561] systemd[1]: Detected architecture x86-64. [ 2.953604] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.979272] systemd[1]: No hostname configured. [ 2.980613] systemd[1]: Set hostname to . [ 2.982371] random: systemd: uninitialized urandom read (16 bytes read) [ 2.984544] systemd[1]: Initializing machine ID from random generator. [ 3.106199] random: systemd: uninitialized urandom read (16 bytes read) [ 3.108487] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.113299] random: systemd: uninitialized urandom read (16 bytes read) [ 3.115821] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.119687] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.628772] device-mapper: uevent: version 1.0.3 [ 3.630674] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.362097] virtio_net virtio0 ens2: renamed from eth0 [ 4.390245] scsi host0: ata_piix [ 4.418876] scsi host1: ata_piix [ 4.420711] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.424216] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.245846] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.399680] random: crng init done [ 9.400935] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.806415] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.941974] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.180854] SELinux: Disabled at runtime. [ 11.242623] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.252090] systemd[1]: Detected virtualization kvm. [ 11.254377] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.711606] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.715638] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.721654] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.726069] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.729226] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.737970] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.741679] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [[ 11.824326] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.233564] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.566966] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.571797] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.690977] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.710338] EDAC sbridge: Ver: 1.1.2 [ 13.902155] Key type dns_resolver registered [ 14.202416] NFS: Registering the id_resolver key type [ 14.204588] Key type id_resolver registered [ 14.206202] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-client login: [ 78.151187] libcfs: loading out-of-tree module taints kernel. [ 78.472619] Key type ._llcrypt registered [ 78.473403] Key type .llcrypt registered [ 78.818257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 78.837969] alg: No test for adler32 (adler32-zlib) [ 80.078099] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 80.971952] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 82.783220] Key type lgssc registered [ 84.252403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 211.611516] Lustre: Mounted lustre-client - version 2.17.58_41_g9380d02 [ 217.977304] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.820373] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing check_logdir /tmp/testlogs/ [ 237.535352] Lustre: lustre-OST0000-osc-ffff9369c56e1000: disconnect after 22s idle [ 239.374980] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing yml_node [ 244.432937] Lustre: DEBUG MARKER: Client: 2.17.58.41 [ 246.875838] Lustre: DEBUG MARKER: MDS: 2.17.58.41 [ 249.224350] Lustre: DEBUG MARKER: OSS: 2.17.58.41 [ 250.690072] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 14 19:18:33 EDT 2026 [ 266.449225] Lustre: Unmounted lustre-client [ 293.498451] Key type lgssc unregistered [ 293.876612] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.880668] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.900313] LNet: Removed LNI 192.168.201.59@tcp [ 294.759238] Key type .llcrypt unregistered [ 294.761366] Key type ._llcrypt unregistered [ 325.068907] Key type ._llcrypt registered [ 325.079823] Key type .llcrypt registered [ 325.340968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 325.362256] alg: No test for adler32 (adler32-zlib) [ 326.276357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 326.367589] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 326.417597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 330.529665] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 330.877471] LNet: 9289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.884286] LNetError: 9289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 330.898721] LNet: Removed LNI 192.168.201.59@tcp [ 331.450562] Key type .llcrypt unregistered [ 331.452391] Key type ._llcrypt unregistered [ 337.733373] Key type ._llcrypt registered [ 337.735874] Key type .llcrypt registered [ 338.912451] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:20:01 (1789428001) [ 339.343444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.359405] alg: No test for adler32 (adler32-zlib) [ 340.276952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.374538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.403429] LNet: 9698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.428553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.511133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 340.542087] LNet: 9705:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 347.934427] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:20:10 (1789428010) [ 348.177635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.211903] LNet: 10284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.245129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.389793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 354.904346] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:20:17 (1789428017) [ 355.115483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.182618] LNet: 10870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.223940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.304153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 361.917932] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:20:24 (1789428024) [ 362.027187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.072236] LNet: 11450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.120571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.302269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 368.659957] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:20:31 (1789428031) [ 368.906630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.936811] LNet: 12039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.965712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.038830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 369.614909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 376.048954] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:20:38 (1789428038) [ 376.230916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.265487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.335787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 376.914186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 377.476824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 384.364587] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:20:46 (1789428046) [ 384.666632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.691994] LNet: 13226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.698692] LNet: 13226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 384.728459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.831686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 390.979933] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:20:53 (1789428053) [ 391.080764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.139247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.234848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 391.290291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 391.731389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 391.754354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 392.066210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 392.119680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.664422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 392.698715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.094157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 393.135375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.635823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 393.698239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.088398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 394.145197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 394.680605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 394.725358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 395.324394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 395.364049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 396.045422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 396.095129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 396.604591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 396.644518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 397.064982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 397.110979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.737850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 397.767167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 403.371817] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:21:05 (1789428065) [ 403.495197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.541537] LNet: 14489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.548056] LNet: 14489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 403.581638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.688831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 403.721521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 409.527728] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:21:11 (1789428071) [ 409.654031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.723078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.819113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.859238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 415.486760] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:21:18 (1789428078) [ 415.548982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.578731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.661149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 415.686748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 416.077476] hrtimer: interrupt took 5481628 ns [ 421.796980] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:21:23 (1789428083) [ 421.914069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.969884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.092322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 422.166629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 429.143538] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:21:31 (1789428091) [ 429.257350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.308588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.378507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 429.411779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 435.550504] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:21:38 (1789428098) [ 435.759853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.790976] LNet: 17425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.797179] LNet: 17425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 435.823865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.894222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 435.924271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 441.932332] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:21:44 (1789428104) [ 442.176395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.216316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.312703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 442.900777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.927788] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:21:51 (1789428111) [ 449.155198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.191067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.428672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 449.592921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.735877] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:21:58 (1789428118) [ 455.803380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.855086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.988946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.178891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.125590] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:22:04 (1789428124) [ 462.332605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.409619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.878138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.068778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.643839] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:22:11 (1789428131) [ 468.829239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.851990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.904678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.939654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.872434] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:22:17 (1789428137) [ 474.948748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.994671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.041762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.074699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.264532] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:22:23 (1789428143) [ 481.591663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.664878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.753414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 487.812801] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:22:30 (1789428150) [ 487.906105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.950700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.047842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 493.716568] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:22:36 (1789428156) [ 493.934949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.964791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.025840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 494.559599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 500.732758] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:22:42 (1789428162) [ 500.945541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.976869] LNet: 23449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.989419] LNet: 23449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 501.029788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.134316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 501.171211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 508.247590] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:22:50 (1789428170) [ 508.351255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.415582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.518663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 508.548307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 515.006576] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:22:57 (1789428177) [ 515.173556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.242187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.300486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 515.350498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 522.335658] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:23:04 (1789428184) [ 522.461411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.503205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.609318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 522.652345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 529.605970] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:23:11 (1789428191) [ 529.798196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.865502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.930144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 530.008580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 530.043356] LNetError: 25807:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 530.099269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 530.129262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 536.453423] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:23:18 (1789428198) [ 536.695663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.732777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.860876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.swWo5O8wWK [ 536.942724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.swWo5O8wWK [ 536.990501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.swWo5O8wWK [ 543.405396] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:23:25 (1789428205) [ 543.659639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.702616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.759721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 543.797162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 543.843911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 549.697654] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 19:23:32 (1789428212) [ 549.897287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.925585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.985683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 550.004698] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 551.186710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 552.920724] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 554.781443] Key type lgssc registered [ 556.665594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 561.539311] LNetError: 27568:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.59@tcp99 has no available nets [ 565.967918] Key type lgssc unregistered [ 566.419232] LNetError: 28675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.443133] LNet: Removed LNI 192.168.201.59@tcp [ 567.047439] Key type .llcrypt unregistered [ 567.051866] Key type ._llcrypt unregistered [ 573.850263] Key type ._llcrypt registered [ 573.858592] Key type .llcrypt registered [ 575.108534] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:23:57 (1789428237) [ 575.607497] Key type .llcrypt unregistered [ 575.609222] Key type ._llcrypt unregistered [ 582.617533] Key type ._llcrypt registered [ 582.619233] Key type .llcrypt registered [ 582.969660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.991588] alg: No test for adler32 (adler32-zlib) [ 583.980663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.080680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.102965] LNet: 29584:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 584.110972] LNet: 29584:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 584.125613] LNet: 29584:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 584.141441] LNet: Message forwarding will be enabled in 2s [ 587.231093] LNet: Message forwarding enabled [ 587.759904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 587.777665] LNet: Message forwarding will stop in 2s to 6s [ 591.330291] LNet: Message forwarding disabled [ 592.350415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 592.390883] LNet: 29683:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 592.398257] LNet: 29683:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 592.407849] LNet: 29683:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 592.431191] LNet: Message forwarding will be enabled in 2s [ 592.651506] LNet: 29696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.218715] Key type .llcrypt unregistered [ 593.220812] Key type ._llcrypt unregistered [ 604.660451] Key type ._llcrypt registered [ 604.662364] Key type .llcrypt registered [ 605.659597] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:24:28 (1789428268) [ 606.086560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.094954] alg: No test for adler32 (adler32-zlib) [ 606.957977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.017455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 612.168496] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:24:34 (1789428274) [ 612.394977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.422640] LNet: 31094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.464335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.573135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 612.608953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 612.640549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 612.674785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 612.705839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 612.734705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 612.777474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 612.819834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 612.876534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 612.948681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 612.983933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 613.056868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 613.096482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 613.138809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 613.197860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 613.236802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 613.283719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 613.341678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 613.389426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 613.411785] LNetError: 31139:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 619.435403] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:24:41 (1789428281) [ 619.585179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.633763] LNet: 31705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.669447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.739623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 626.502252] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:24:48 (1789428288) [ 628.263692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg159-server [ 630.128211] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:24:52 (1789428292) [ 630.228276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.262470] LNet: 32635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.299920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.378281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 630.403230] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 631.049435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[60-64]@tcp [ 631.079406] LNet: 32775:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 631.122933] LNetError: 32645:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 631.143150] LNetError: 32645:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.60@tcp has gone from up to down [ 634.210434] LNetError: 32652:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.62@tcp has gone from up to down [ 637.523766] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:24:59 (1789428299) [ 639.372888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg159-server [ 641.001501] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:25:03 (1789428303) [ 641.246387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.279314] LNet: 33711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.290621] LNetError: 33711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.312426] LNet: Removed LNI 192.168.201.59@tcp [ 641.344436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.406190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.425331] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 641.941304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 641.957735] LNet: 33828:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 641.964272] LNet: 33828:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 641.984701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 647.681906] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:25:09 (1789428309) [ 647.781715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.804333] LNet: 34414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.822553] LNetError: 34414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 647.854834] LNet: Removed LNI 192.168.201.59@tcp [ 647.920361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.015765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 648.110493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.160336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.243478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.285745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.321662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.377923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 648.457206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.488913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.544320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 648.609291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.641451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.719800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 648.842434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.883641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.991771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.095703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.148599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.208649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 649.295605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.349275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.415609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.514536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.560482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.647205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 649.744643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.789694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.854129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.944892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.018989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.090603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 650.137784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.177827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.241147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.989066] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:25:18 (1789428318) [ 658.183224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg159-server [ 659.693981] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:25:22 (1789428322) [ 661.168424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg159-server [ 662.324775] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:25:24 (1789428324) [ 662.469748] LNet: 35822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.472868] LNet: 35822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 662.952458] Key type .llcrypt unregistered [ 662.955239] Key type ._llcrypt unregistered [ 667.979504] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.199526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.215314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.230138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.841266] Key type ._llcrypt registered [ 670.845119] Key type .llcrypt registered [ 671.193444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 671.220547] alg: No test for adler32 (adler32-zlib) [ 672.209578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.323798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.344085] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 672.996649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 673.017468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 674.883929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 674.906681] LNet: Removed LNI 10.1.2.3@tcp [ 675.021423] LNet: 36534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.026666] LNetError: 36534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.038321] LNet: Removed LNI 192.168.201.59@tcp [ 675.671522] Key type .llcrypt unregistered [ 675.673729] Key type ._llcrypt unregistered [ 686.252100] Key type ._llcrypt registered [ 686.254497] Key type .llcrypt registered [ 687.605166] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:25:49 (1789428349) [ 688.044503] Key type .llcrypt unregistered [ 688.045516] Key type ._llcrypt unregistered [ 694.585307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.779047] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.782933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.786213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 697.399158] Key type ._llcrypt registered [ 697.400470] Key type .llcrypt registered [ 697.635439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.658780] alg: No test for adler32 (adler32-zlib) [ 698.580070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.637249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.657533] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 699.341249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 699.385491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 702.234451] LNet: 38023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.239675] LNetError: 38023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 702.254870] LNet: Removed LNI 10.1.2.3@tcp [ 702.268746] LNet: Removed LNI 192.168.201.59@tcp [ 703.097450] Key type .llcrypt unregistered [ 703.101412] Key type ._llcrypt unregistered [ 715.551462] Key type ._llcrypt registered [ 715.554434] Key type .llcrypt registered [ 716.853547] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:26:19 (1789428379) [ 717.484283] Key type .llcrypt unregistered [ 717.485805] Key type ._llcrypt unregistered [ 723.064079] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.347184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.366154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.373501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.389163] Key type ._llcrypt registered [ 726.390629] Key type .llcrypt registered [ 726.719162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.752989] alg: No test for adler32 (adler32-zlib) [ 727.950546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.099232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 728.162984] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 730.778712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.808227] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.812446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 733.857715] LNet: Removed LNI 10.1.2.3@tcp [ 733.906078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 733.980472] LNet: Removed LNI 10.1.2.31@tcp [ 734.018250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 734.081664] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 735.283974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 735.313138] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 737.256779] LNet: 39940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.266947] LNetError: 39940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 737.277581] LNet: Removed LNI 10.1.2.31@tcp [ 737.289493] LNet: Removed LNI 10.1.2.3@tcp [ 738.368184] Key type .llcrypt unregistered [ 738.371882] Key type ._llcrypt unregistered [ 753.210575] Key type ._llcrypt registered [ 753.213259] Key type .llcrypt registered [ 754.507633] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:26:56 (1789428416) [ 754.790645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.802755] alg: No test for adler32 (adler32-zlib) [ 755.700771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.763864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.791443] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 756.434774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.451188] LNet: 40892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.460389] LNetError: 40892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.477977] LNet: Removed LNI 192.168.201.59@tcp [ 756.492698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.536275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.59@tcp [ 756.551765] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 757.167754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.59@tcp [ 757.208681] LNet: Removed LNI 192.168.201.59@tcp [ 757.313835] LNet: 41027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.110551] Key type .llcrypt unregistered [ 758.114981] Key type ._llcrypt unregistered [ 770.104982] Key type ._llcrypt registered [ 770.108274] Key type .llcrypt registered [ 771.774697] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:27:13 (1789428433) [ 773.310447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg159-server [ 774.825600] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:27:17 (1789428437) [ 775.209333] Key type .llcrypt unregistered [ 775.212615] Key type ._llcrypt unregistered [ 780.765066] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.925890] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.934686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.941626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.512957] Key type ._llcrypt registered [ 783.515293] Key type .llcrypt registered [ 783.717547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.745225] alg: No test for adler32 (adler32-zlib) [ 784.645452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.703576] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 785.369850] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 787.171095] LNet: 42859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.178627] LNetError: 42859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.181058] LNet: Removed LNI 192.168.201.59@tcp1 [ 787.193346] LNet: Removed LNI 10.1.2.3@tcp2 [ 787.675391] Key type .llcrypt unregistered [ 787.677606] Key type ._llcrypt unregistered [ 798.589169] Key type ._llcrypt registered [ 798.594444] Key type .llcrypt registered [ 800.122456] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:27:42 (1789428462) [ 800.574210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 800.609861] alg: No test for adler32 (adler32-zlib) [ 801.595240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.672682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.688788] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 802.656972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.689232] LNet: 43825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.696561] LNetError: 43825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.727171] LNet: Removed LNI 192.168.201.59@tcp [ 802.769879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.856390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 802.888753] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 809.939162] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:27:52 (1789428472) [ 810.122374] LNet: 44412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.132950] LNetError: 44412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.166110] LNet: Removed LNI 192.168.201.59@tcp [ 810.784942] Key type .llcrypt unregistered [ 810.789048] Key type ._llcrypt unregistered [ 817.391661] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.519120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.525057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.531232] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.176797] Key type ._llcrypt registered [ 820.179714] Key type .llcrypt registered [ 820.423645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.438348] alg: No test for adler32 (adler32-zlib) [ 821.395666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.457314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.489860] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 822.218310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.243563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.449467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.469677] LNet: 45155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.475699] LNetError: 45155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.480821] LNet: Removed LNI 10.1.2.3@tcp [ 824.490250] LNet: Removed LNI 192.168.201.59@tcp [ 824.515295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.612657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.648204] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 824.652542] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.305452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.350518] LNet: 45182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.357167] LNetError: 45182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 825.367219] LNetError: 45182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 825.379965] LNet: Removed LNI 10.1.2.3@tcp [ 825.399783] LNet: Removed LNI 192.168.201.59@tcp [ 825.443107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.512632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.554297] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 825.560425] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.229620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 826.259662] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 827.225650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 827.246426] LNet: 45344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.254871] LNetError: 45344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 827.260071] LNetError: 45344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 827.265928] LNet: Removed LNI 10.1.2.3@tcp [ 827.267535] LNet: Removed LNI 192.168.201.59@tcp [ 827.278607] LNet: Removed LNI 192.168.201.59@tcp2 [ 827.295865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.352817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 827.376603] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 827.382487] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.389445] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 834.409349] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:28:16 (1789428496) [ 834.700766] LNet: 45940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.710400] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 834.720200] LNetError: 45940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 834.730450] LNet: Removed LNI 10.1.2.3@tcp [ 834.732602] LNet: Removed LNI 192.168.201.59@tcp [ 834.759867] LNet: Removed LNI 192.168.201.59@tcp2 [ 835.372804] Key type .llcrypt unregistered [ 835.382385] Key type ._llcrypt unregistered [ 841.674749] Key type ._llcrypt registered [ 841.676581] Key type .llcrypt registered [ 841.877391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.894683] alg: No test for adler32 (adler32-zlib) [ 842.800988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.836674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 842.861094] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 843.604182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 843.618505] LNet: 46331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.623425] LNetError: 46331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.638765] LNet: Removed LNI 192.168.201.59@tcp [ 843.651550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.725909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 843.756362] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 844.149378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.164640] LNet: 46361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.169108] LNetError: 46361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.183256] LNet: Removed LNI 192.168.201.59@tcp [ 844.211472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.293078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.326462] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 844.783857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.812295] LNet: Removed LNI 192.168.201.59@tcp [ 844.837089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.891271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.924103] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 845.503812] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 845.680329] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 845.688934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.692375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.307546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.340815] LNet: 46469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.347178] LNet: 46469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 848.358389] LNetError: 46469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.361148] LNetError: 46469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 848.371497] LNet: Removed LNI 192.168.201.59@tcp [ 848.406675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.485911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 848.504550] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 855.808886] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:28:37 (1789428517) [ 855.899099] LNet: 47057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.906302] LNetError: 47057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.940253] LNet: Removed LNI 192.168.201.59@tcp [ 856.803183] Key type .llcrypt unregistered [ 856.804767] Key type ._llcrypt unregistered [ 862.927219] Key type ._llcrypt registered [ 862.929152] Key type .llcrypt registered [ 863.129212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.151976] alg: No test for adler32 (adler32-zlib) [ 864.024843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.103567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.120141] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 864.752348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.768772] LNet: 47461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.776809] LNetError: 47461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.789801] LNet: Removed LNI 192.168.201.59@tcp [ 864.805203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.881990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 864.915614] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 871.539700] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:28:53 (1789428533) [ 871.729267] LNet: 48051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.733428] LNetError: 48051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.758338] LNet: Removed LNI 192.168.201.59@tcp [ 872.438510] Key type .llcrypt unregistered [ 872.440333] Key type ._llcrypt unregistered [ 879.145186] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 879.406578] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 879.420638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 879.432060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 882.005454] Key type ._llcrypt registered [ 882.008323] Key type .llcrypt registered [ 882.176355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 882.195251] alg: No test for adler32 (adler32-zlib) [ 883.145247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.223550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 883.247191] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 884.197726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.215838] LNet: 48501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.222441] LNetError: 48501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.242982] LNet: Removed LNI 192.168.201.59@tcp [ 884.262300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.315376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 884.332712] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 890.145839] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:29:12 (1789428552) [ 890.387546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.410410] LNet: 49099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.415304] LNetError: 49099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.430242] LNet: Removed LNI 192.168.201.59@tcp [ 890.445564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.544192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 896.231302] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:29:18 (1789428558) [ 896.305200] LNet: 49674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.709756] Key type .llcrypt unregistered [ 896.712950] Key type ._llcrypt unregistered [ 901.027551] Key type ._llcrypt registered [ 901.028661] Key type .llcrypt registered [ 901.146788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.155959] alg: No test for adler32 (adler32-zlib) [ 902.015239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.052502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.069197] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 902.763485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.779521] LNet: 50075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.784205] LNetError: 50075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.810503] LNet: Removed LNI 192.168.201.59@tcp [ 902.845265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.929828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 902.951623] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 909.205336] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:29:31 (1789428571) [ 909.361561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.376701] LNet: 50670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.382394] LNetError: 50670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.396417] LNet: Removed LNI 192.168.201.59@tcp [ 909.431919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.502712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 916.202993] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:29:38 (1789428578) [ 917.983957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 920.492338] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:29:42 (1789428582) [ 922.164633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 925.346036] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:29:46 (1789428586) [ 926.914915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 928.774721] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:29:50 (1789428590) [ 930.269789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 931.880795] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:29:54 (1789428594) [ 933.796051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 935.722362] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:29:57 (1789428597) [ 935.854095] LNet: 52690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.610550] Key type .llcrypt unregistered [ 936.614798] Key type ._llcrypt unregistered [ 942.766569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.038931] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.047094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.052698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.696840] Key type ._llcrypt registered [ 945.698976] Key type .llcrypt registered [ 945.893288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.936327] alg: No test for adler32 (adler32-zlib) [ 946.930724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.005764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.023282] LNet: Added LNI 192.168.201.59@tcp [8/512/0/180] [ 947.036656] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 947.039544] LNet: 53012:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.143514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.161482] LNet: 53031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.166528] LNetError: 53031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.177666] LNet: Removed LNI 10.1.2.3@tcp [ 947.181298] LNetError: 53031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.194372] LNet: Removed LNI 192.168.201.59@tcp [ 947.225863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.290949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.325463] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 947.433677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.493270] LNet: 53055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.504793] LNetError: 53055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.523836] LNet: Removed LNI 192.168.201.59@tcp [ 947.553413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.673342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.708882] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 947.723396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.725034] LNet: 53063:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.858913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.886448] LNet: 53081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.893293] LNetError: 53081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.900248] LNet: Removed LNI 10.1.2.3@tcp [ 947.905100] LNetError: 53081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.922407] LNet: Removed LNI 192.168.201.59@tcp [ 947.957724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.051465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.082827] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 948.098470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 948.108390] LNet: 53089:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.329892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.347486] LNet: 53115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.352464] LNetError: 53115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.357520] LNet: Removed LNI 10.1.2.3@tcp [ 948.360334] LNetError: 53115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.378246] LNet: Removed LNI 192.168.201.59@tcp [ 948.407506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.473638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.487791] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 948.497511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.595280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.628269] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.634809] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.647070] LNet: Removed LNI 10.1.2.3@tcp [ 948.648434] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.657437] LNet: Removed LNI 192.168.201.59@tcp [ 948.670046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.728969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.764702] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 948.779106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.780572] LNet: 53151:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.877477] LNet: Removed LNI 10.1.2.3@tcp [ 948.891325] LNet: Removed LNI 192.168.201.59@tcp [ 949.548837] Key type .llcrypt unregistered [ 949.550626] Key type ._llcrypt unregistered [ 961.708525] Key type ._llcrypt registered [ 961.711312] Key type .llcrypt registered [ 962.892623] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:30:25 (1789428625) [ 964.483319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 966.079356] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:30:28 (1789428628) [ 966.538415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 966.555347] alg: No test for adler32 (adler32-zlib) [ 967.442220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.484046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.501051] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 968.182971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.59 [ 968.199696] LNet: 54412:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 968.210306] LNet: Removed LNI 192.168.201.59@tcp [ 974.196622] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:30:36 (1789428636) [ 974.279335] LNet: 54973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.288638] LNetError: 54973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.325942] LNet: Removed LNI 192.168.201.59@tcp [ 974.845727] Key type .llcrypt unregistered [ 974.850779] Key type ._llcrypt unregistered [ 981.301910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.754835] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.602385] Key type ._llcrypt registered [ 984.606850] Key type .llcrypt registered [ 984.829836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.846488] alg: No test for adler32 (adler32-zlib) [ 986.269969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.101437] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:30:54 (1789428654) [ 992.369071] LNet: 55892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.375609] LNetError: 55892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.406713] LNet: Removed LNI 10.1.2.3@tcp [ 992.953433] Key type .llcrypt unregistered [ 992.955814] Key type ._llcrypt unregistered [ 998.903119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.292619] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.296122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.299938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1000.830486] Key type ._llcrypt registered [ 1000.835206] Key type .llcrypt registered [ 1001.064861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.076552] alg: No test for adler32 (adler32-zlib) [ 1002.188278] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1008.265850] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:31:10 (1789428670) [ 1009.496685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1011.146616] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:31:13 (1789428673) [ 1011.269964] LNet: 57060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.283988] LNetError: 57060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1011.323158] LNet: Removed LNI 10.1.2.3@tcp [ 1011.798571] Key type .llcrypt unregistered [ 1011.800063] Key type ._llcrypt unregistered [ 1017.318845] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1017.632558] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1017.634684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1017.645976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1020.313371] Key type ._llcrypt registered [ 1020.315716] Key type .llcrypt registered [ 1020.636332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.653738] alg: No test for adler32 (adler32-zlib) [ 1021.602680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.683296] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1030.552978] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:31:33 (1789428693) [ 1030.964430] LNet: 58292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.967584] LNetError: 58292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1030.974767] LNet: Removed LNI 10.1.2.3@tcp [ 1031.347449] Key type .llcrypt unregistered [ 1031.350334] Key type ._llcrypt unregistered [ 1038.772885] Key type ._llcrypt registered [ 1038.774124] Key type .llcrypt registered [ 1038.919952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1038.932845] alg: No test for adler32 (adler32-zlib) [ 1039.983506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1040.006626] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1046.271087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1047.180877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1047.250593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1047.281360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1047.309450] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1047.502558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.516460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1047.532438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.549973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.564562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1047.587599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.606314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.642473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.678670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.692678] LNet: 58728:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.710856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.740752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.775285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.808225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.832985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.854023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.874779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1048.426032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1048.456619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1048.475453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1048.495245] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1048.655517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.674854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1048.702603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.727866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.746619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1048.782777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.828335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.869878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.901545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.915371] LNet: 58839:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.931179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.963932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.994892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1049.026915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1049.054391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.083345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.119458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1049.727400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1049.759672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1049.788921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1049.803352] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1049.963978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.982059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1050.018572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.053794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.072178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1050.088992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.106416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.124285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.142768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.153410] LNet: 58950:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.162479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.198534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.248989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.278959] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1050.289864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.343909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.382928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.406628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.126310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1051.173345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1051.228596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1051.567569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.631916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1051.669618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.718843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.754901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1051.791569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.831679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.885520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.923860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.950705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.977768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.003581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.032725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.063444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.081786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.115819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.582213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1052.621098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1052.649738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1052.663581] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1052.667455] LNet: Skipped 2 previous similar messages [ 1052.868981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.902538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1052.967502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.025713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.084405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1053.145552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.187797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.229173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.279513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.318655] LNet: 59173:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.327441] LNet: 59173:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1053.357358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.395578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.447934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.505220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.601638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.663182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.729770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.793636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1054.874607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1054.966326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1055.303915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.335258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.383409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.429319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.456765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1055.480741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.518382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.562776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.599882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.643601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.679625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.719659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.763423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.837186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.947305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.992229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.471556] LNet: 59325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.476647] LNetError: 59325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.509311] LNet: Removed LNI 192.168.201.59@tcp [ 1057.079678] Key type .llcrypt unregistered [ 1057.082805] Key type ._llcrypt unregistered [ 1077.421358] Key type ._llcrypt registered [ 1077.424353] Key type .llcrypt registered [ 1078.761073] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:32:20 (1789428740) [ 1079.573437] Key type .llcrypt unregistered [ 1079.577935] Key type ._llcrypt unregistered [ 1085.937420] Key type ._llcrypt registered [ 1085.939951] Key type .llcrypt registered [ 1086.094562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1086.111288] alg: No test for adler32 (adler32-zlib) [ 1087.001485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.021107] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1094.757199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1108.126976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.187758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1108.220592] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1109.902587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1109.938848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1109.979567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1110.007376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1110.036630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1110.068262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1110.108616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1110.148531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1110.188627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1110.209717] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1110.215711] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1110.484882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.513414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.538432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.571804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.596587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.623144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.647669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.677431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.708798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.723709] LNet: 60791:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.736590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.783376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.838662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.878727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.910596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.947761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.026678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.732587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1111.765513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1111.797174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1111.823687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1111.839934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1111.862139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1111.895820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1111.932278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1111.973253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1111.998866] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1112.005775] LNet: Skipped 2 previous similar messages [ 1112.311134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.366655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1112.419950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.474560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.536156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1112.593714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.651865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.706944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.743348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.767750] LNet: 60914:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.799129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.835340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.879571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.912984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.964974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.036312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.060959] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1113.063665] LNet: Skipped 2 previous similar messages [ 1113.084339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.789517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1113.815734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1113.845464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1113.879540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1113.924155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1113.961704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1113.993760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1114.022552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1114.059156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1114.399940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.443838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.491473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.559993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.602814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1114.658589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.759226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.834482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.883253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.897403] LNet: 61038:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.937279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.023473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.085340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.111344] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1115.114319] LNet: Skipped 3 previous similar messages [ 1115.139103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.204526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.255094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.301469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.058648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1116.091909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1116.117665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1116.146643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1116.174323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1116.204305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1116.229780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1116.266560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1116.303633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1116.580267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.603981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.636351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.680693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.706506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1116.733594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.758611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.780550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.816961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.848610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.888575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.932766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.970032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.999380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.030156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.066479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1117.800831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1117.842962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1117.885453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1117.930861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1118.018040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1118.091547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1118.138771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1118.175272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1118.207958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1118.480587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.543342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.604516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.649475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.684277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1118.712425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.779939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.834630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.879676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.910954] LNet: 61285:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.924962] LNet: 61285:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1118.957043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.003546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.071196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.140159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.185780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.241692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.294411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.801723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1119.859973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1119.896210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1119.962754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1120.022396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1120.098733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1120.168398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1120.196918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1120.238588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1120.270542] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1120.276037] LNet: Skipped 6 previous similar messages [ 1120.575737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.605525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.646442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.678374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.715775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1120.739550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.781728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.810502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.871309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.941285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.996291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.073274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.116699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.179391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.239853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.323421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.283422] LNet: 61484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.289315] LNetError: 61484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.295785] LNet: Removed LNI 192.168.201.59@tcp [ 1128.358977] LNet: Removed LNI 192.168.201.59@tcp1 [ 1128.803606] Key type .llcrypt unregistered [ 1128.806226] Key type ._llcrypt unregistered [ 1146.101974] Key type ._llcrypt registered [ 1146.105402] Key type .llcrypt registered [ 1148.136315] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:33:29 (1789428809) [ 1149.092242] Key type .llcrypt unregistered [ 1149.093146] Key type ._llcrypt unregistered [ 1156.933715] Key type ._llcrypt registered [ 1156.936037] Key type .llcrypt registered [ 1157.234374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1157.259006] alg: No test for adler32 (adler32-zlib) [ 1158.335371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1158.364980] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1166.189548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1166.631928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1166.660606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1166.693816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1166.720664] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1166.967990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.998604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.030724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.065491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.093277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1167.132429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.168787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.201731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.239937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.279508] LNet: 62732:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.302734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.349658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.390358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.433572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.459606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.486881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.514549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.039831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1168.081747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1168.124549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1168.144338] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1168.444655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.489780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.520592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.575209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.637172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1168.706768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.772738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.843455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.898583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.919725] LNet: 62843:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.945829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.994440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.029953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.097466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.132465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.164409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.210665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.899818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1169.936723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1169.970588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1169.989810] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1170.200626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.225352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.252782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.277418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.306431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1170.346769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.375714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.408884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.443584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.462688] LNet: 62954:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.478415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.509293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.537422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.568853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.625101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.684565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.752254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1171.126868] LNet: 62995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.140266] LNetError: 62995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.167217] LNet: Removed LNI 192.168.201.59@tcp [ 1171.658753] Key type .llcrypt unregistered [ 1171.671417] Key type ._llcrypt unregistered [ 1189.964668] Key type ._llcrypt registered [ 1189.965781] Key type .llcrypt registered [ 1191.451664] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:34:13 (1789428853) [ 1192.457474] Key type .llcrypt unregistered [ 1192.463336] Key type ._llcrypt unregistered [ 1200.219238] Key type ._llcrypt registered [ 1200.222884] Key type .llcrypt registered [ 1200.438468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.450569] alg: No test for adler32 (adler32-zlib) [ 1201.473909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1201.501370] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1210.627935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1221.442609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.520655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1221.553711] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1223.232508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1223.291648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1223.365716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1223.439088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1223.490573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1223.565614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1223.596405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1223.635156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1223.684440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1223.702709] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1223.711507] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1224.033476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.080438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.160857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.227830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.318352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1224.366574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.417338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.531498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.601534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.630485] LNet: 64459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.652170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.714274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.735456] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1224.747453] LNet: Skipped 3 previous similar messages [ 1224.769974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.821957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.863257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.917792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.974835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.012540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.692271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1225.723787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1225.743770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1225.764614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1225.791288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1225.828696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1225.864944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1225.899774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1225.968761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1225.999503] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1226.301427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.347706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.416578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.487791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.518203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1226.569628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.602560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.650832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.712651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.755621] LNet: 64585:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.794220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.837638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.895525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.932693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.971603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.018241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.047247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1227.073575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.554921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1227.579899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1227.610402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1227.640411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1227.671605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1227.716422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1227.757781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1227.794785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1227.832792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1228.100282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.132438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.162503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.188576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.215063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1228.242223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.269649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.306541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.345104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.366685] LNet: 64710:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.383928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.456129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.508535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.552722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.603543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.676986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.694719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1228.711839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.723411] LNet: 64788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.736938] LNetError: 64788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.747469] LNet: Removed LNI 192.168.201.59@tcp [ 1233.763442] LNet: Removed LNI 192.168.201.59@tcp1 [ 1234.260542] Key type .llcrypt unregistered [ 1234.264896] Key type ._llcrypt unregistered [ 1251.727211] Key type ._llcrypt registered [ 1251.731279] Key type .llcrypt registered [ 1253.277202] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:35:15 (1789428915) [ 1255.263136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1256.611816] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:35:19 (1789428919) [ 1257.407339] Key type .llcrypt unregistered [ 1257.409116] Key type ._llcrypt unregistered [ 1263.032203] Key type ._llcrypt registered [ 1263.036400] Key type .llcrypt registered [ 1263.447788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1263.461785] alg: No test for adler32 (adler32-zlib) [ 1264.436741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1264.471755] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1271.707325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1272.699586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1272.728511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1272.763402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1272.791462] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1272.936751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1272.950314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1272.967561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1272.983939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1273.009494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1273.033899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1273.071421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1273.118696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1273.147522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1273.160621] LNet: 66329:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1273.171345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1273.194563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1273.219933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1273.256684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.288662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1273.321436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.565918] LNet: 66368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.572161] LNetError: 66368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.588051] LNet: Removed LNI 192.168.201.59@tcp [ 1273.979763] Key type .llcrypt unregistered [ 1273.981751] Key type ._llcrypt unregistered [ 1298.961138] Key type ._llcrypt registered [ 1298.964704] Key type .llcrypt registered [ 1299.123530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1299.140202] alg: No test for adler32 (adler32-zlib) [ 1300.094774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1300.129780] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1308.344661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1320.783564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1320.830443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1320.866623] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1322.646440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1322.710542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1322.755386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1322.799408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1322.841808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1322.885670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1322.930585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1322.980707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1323.031824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1323.065927] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1323.494289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1323.570162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1323.607552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1323.650645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1323.681905] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1323.703653] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1323.755376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1323.786602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1323.844907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1323.888764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1323.930773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1323.948849] LNet: 67246:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1323.985548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1324.052040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1324.111030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1324.166964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1324.199601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1324.243489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1329.783165] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1330.283955] LNet: 67320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.289843] LNetError: 67320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.300912] LNet: Removed LNI 192.168.201.59@tcp [ 1330.318056] LNet: Removed LNI 192.168.201.59@tcp1 [ 1330.811478] Key type .llcrypt unregistered [ 1330.813241] Key type ._llcrypt unregistered [ 1349.555518] Key type ._llcrypt registered [ 1349.558771] Key type .llcrypt registered [ 1351.060263] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:36:53 (1789429013) [ 1351.547312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1351.585738] alg: No test for adler32 (adler32-zlib) [ 1352.576425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.636928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1352.658090] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1353.517885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1353.534678] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1354.346316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1354.437817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1354.478067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1354.511978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1354.542887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1354.576963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1354.613836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1354.639304] LNetError: 68416:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1354.657622] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1354.658665] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1356.708398] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1357.136068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1360.801774] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1361.871142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.942938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.042455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.065948] LNet: 68523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.069768] LNetError: 68523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.073776] LNet: Removed LNI 192.168.201.59@tcp [ 1362.082773] LNet: Removed LNI 192.168.201.59@tcp1 [ 1362.127587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.218950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.238885] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1363.018688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.037817] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1364.266895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1364.362831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.460482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1364.582761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1364.631914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1364.690867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1364.731694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1364.786376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1364.822444] LNetError: 68796:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1365.280506] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1368.564522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1371.423667] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1371.438087] LNet: Skipped 1 previous similar message [ 1376.848108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1376.939837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1376.973542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.442283] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:37:27 (1789429047) [ 1385.884149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.925912] LNet: 69515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.932884] LNetError: 69515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.949793] LNetError: 69515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1385.960767] LNet: Removed LNI 192.168.201.59@tcp [ 1385.974870] LNet: Removed LNI 192.168.201.59@tcp1 [ 1386.052837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.138697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.178905] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1386.996405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1387.016187] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1388.205579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1388.290466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1388.318414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1388.379512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1388.429142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1388.489513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1388.550976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1388.687615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.215734] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.59@tcp [ 1389.217338] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1389.231533] LNet: Skipped 1 previous similar message [ 1404.448672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1404.650436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.700269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1404.735312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 500 [ 1404.830237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1411.739425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1411.799993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.823765] LNet: 70006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.829875] LNetError: 70006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.834962] LNetError: 70006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1411.850365] LNet: Removed LNI 192.168.201.59@tcp [ 1411.869427] LNet: Removed LNI 192.168.201.59@tcp1 [ 1411.932859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.990986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.030288] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1413.107836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.131410] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1414.108685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1414.146328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.241689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1414.278227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1414.355046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1414.402255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1414.453891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.508222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1415.071516] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1415.084896] LNet: Skipped 3 previous similar messages [ 1416.034635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.629259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.676207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1426.738905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1435.731978] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:38:17 (1789429097) [ 1436.265552] LNet: 71053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.280343] LNetError: 71053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.285763] LNetError: 71053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1437.343210] LNet: Removed LNI 192.168.201.59@tcp [ 1437.358356] LNet: Removed LNI 192.168.201.59@tcp1 [ 1438.297676] Key type .llcrypt unregistered [ 1438.301411] Key type ._llcrypt unregistered [ 1446.554946] Key type ._llcrypt registered [ 1446.557499] Key type .llcrypt registered [ 1446.841485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1446.884733] alg: No test for adler32 (adler32-zlib) [ 1447.814736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1447.840933] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1458.674916] Lustre: DEBUG MARKER: Initial discovery [ 1458.922025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1462.010453] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1462.242366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1462.360541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1462.398644] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1462.450281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1462.494508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -l 3 [ 1462.572242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -e local_error [ 1462.626270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m PUT -l 6 [ 1463.203576] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1464.703952] Lustre: DEBUG MARKER: Force oleg159-client.virtnet to discover 192.168.201.159@tcp (in background) [ 1464.799375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1466.978606] Lustre: DEBUG MARKER: Force oleg159-server to discover 192.168.201.59@tcp [ 1467.935263] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1470.370602] Lustre: DEBUG MARKER: Wait for 71805 [ 1471.519373] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1472.691393] Lustre: DEBUG MARKER: Finished wait on 71805 [ 1472.766766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1472.823747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1473.138544] LNet: 72265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.145718] LNetError: 72265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.173786] LNet: Removed LNI 192.168.201.59@tcp [ 1473.763431] Key type .llcrypt unregistered [ 1473.765583] Key type ._llcrypt unregistered [ 1494.670970] Key type ._llcrypt registered [ 1494.673952] Key type .llcrypt registered [ 1495.968585] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:39:18 (1789429158) [ 1496.841438] Key type .llcrypt unregistered [ 1496.845563] Key type ._llcrypt unregistered [ 1504.122368] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1504.330697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1504.339684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1504.354536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1507.093437] Key type ._llcrypt registered [ 1507.096567] Key type .llcrypt registered [ 1507.309320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.323908] alg: No test for adler32 (adler32-zlib) [ 1508.288155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.387692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.410447] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1509.243476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1509.262220] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1519.031238] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:39:40 (1789429180) [ 1519.229778] LNet: 74330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.238730] LNetError: 74330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1519.251523] LNet: Removed LNI 10.1.2.3@tcp [ 1519.272926] LNet: Removed LNI 192.168.201.59@tcp [ 1520.155497] Key type .llcrypt unregistered [ 1520.159879] Key type ._llcrypt unregistered [ 1528.480976] Key type ._llcrypt registered [ 1528.490924] Key type .llcrypt registered [ 1528.700939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.729045] alg: No test for adler32 (adler32-zlib) [ 1529.836196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.886173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1529.915053] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1530.756852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1530.778904] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1531.976067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.59@tcp [ 1533.087445] LNetError: Unexpected error -22 connecting to 192.168.201.59@tcp at host 192.168.201.59:988 [ 1539.892496] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:40:01 (1789429201) [ 1540.226093] LNet: 75467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.248909] LNetError: 75467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1541.279252] LNet: Removed LNI 10.1.2.3@tcp [ 1541.280813] LNetError: 75467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.304668] LNet: Removed LNI 192.168.201.59@tcp [ 1542.147388] Key type .llcrypt unregistered [ 1542.152624] Key type ._llcrypt unregistered [ 1549.947233] Key type ._llcrypt registered [ 1549.950074] Key type .llcrypt registered [ 1550.265164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1550.284746] alg: No test for adler32 (adler32-zlib) [ 1551.447863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.566818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1551.606631] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1553.005424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1553.031761] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1554.355119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.59@tcp1 --nid 192.168.201.59@tcp2 [ 1560.688187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp2 [ 1560.721562] LNetError: 76312:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.59@tcp1 to 192.168.201.59@tcp2 - Route table may be misconfigured [ 1560.750515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp2 192.168.201.59@tcp1 [ 1571.816394] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:40:33 (1789429233) [ 1572.137503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.191642] LNet: 76883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.218479] LNetError: 76883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.281105] LNet: Removed LNI 192.168.201.59@tcp1 [ 1573.296627] LNetError: 76883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.318947] LNet: Removed LNI 192.168.201.59@tcp2 [ 1573.389223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.461492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.489394] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1574.392406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1574.406621] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1575.449158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1575.494191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.59@tcp [ 1575.523622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp1 [ 1575.565120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp -e network_timeout [ 1575.594236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -e network_timeout [ 1575.655461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -e network_timeout [ 1575.715401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -e network_timeout [ 1575.733391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1575.746731] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1575.763820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1575.828180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1576.231863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1576.485146] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1585.729744] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:40:47 (1789429247) [ 1586.163905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.231776] LNet: 77747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.241567] LNetError: 77747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.256090] LNet: Removed LNI 192.168.201.59@tcp [ 1587.310444] LNet: Removed LNI 192.168.201.59@tcp1 [ 1587.371807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.511298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1588.154727] LNet: 77784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.958348] Key type .llcrypt unregistered [ 1588.966623] Key type ._llcrypt unregistered [ 1604.299397] Key type ._llcrypt registered [ 1604.303550] Key type .llcrypt registered [ 1605.836879] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:41:07 (1789429267) [ 1606.288167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1606.302868] alg: No test for adler32 (adler32-zlib) [ 1607.373272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.369471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1611.397912] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:41:13 (1789429273) [ 1611.600254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1611.651667] LNet: 78880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1611.678300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.754890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1611.781231] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1612.797954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1612.822211] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1614.200681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1614.240482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp1 [ 1614.290676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp1 [ 1623.363769] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:41:24 (1789429284) [ 1626.319412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg159-server [ 1628.934480] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:41:30 (1789429290) [ 1630.829853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg159-server [ 1633.122480] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:41:35 (1789429295) [ 1635.936387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg159-server [ 1638.393188] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:41:40 (1789429300) [ 1642.022871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg159-server [ 1644.184479] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:41:46 (1789429306) [ 1647.202257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg159-server [ 1650.922598] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:41:51 (1789429311) [ 1653.248948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg159-server [ 1655.940646] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:41:57 (1789429317) [ 1658.811041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg159-server [ 1661.414573] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:42:02 (1789429322) [ 1663.935603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg159-server [ 1666.134654] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:42:08 (1789429328) [ 1669.326541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg159-server [ 1672.162151] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:42:13 (1789429333) [ 1672.315203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.371373] LNet: 82755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.387809] LNetError: 82755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.394418] LNet: Removed LNI 192.168.201.59@tcp [ 1673.454796] LNet: Removed LNI 192.168.201.59@tcp1 [ 1673.558825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.768134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.856510] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1675.267495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1675.387706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1675.602550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.629386] LNet: 82909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.635398] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.641112] LNetError: 82909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1675.662120] LNet: Removed LNI 192.168.201.59@tcp [ 1675.710794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.781722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1675.806626] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1676.785657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1677.016216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.036347] LNetError: 83055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.054898] LNet: Removed LNI 192.168.201.59@tcp [ 1678.105624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.264695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.333854] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1679.358103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1679.472939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1679.743637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.775749] LNet: 83202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.796518] LNet: 83202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1679.802230] LNetError: 83202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.833959] LNet: Removed LNI 192.168.201.59@tcp [ 1679.899456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.029668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1680.081401] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1681.442087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1681.635212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.728414] LNet: Removed LNI 192.168.201.59@tcp [ 1682.798348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.933258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.985994] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1684.202599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1684.362692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1685.839491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.908871] LNet: 83504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.919315] LNet: 83504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1685.953300] LNetError: 83504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.965789] LNetError: 83504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1687.022392] LNet: Removed LNI 192.168.201.59@tcp [ 1687.075207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.185817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1687.247950] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1688.297439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1688.710748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.767135] LNet: Removed LNI 192.168.201.59@tcp [ 1689.801774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.869678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.893891] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1690.758300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1690.840788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1691.156919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.268748] LNet: Removed LNI 192.168.201.59@tcp [ 1692.339458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.466230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1692.509199] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1693.689046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1693.999272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.029678] LNet: 83942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.038495] LNet: 83942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1694.045450] LNetError: 83942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.057134] LNetError: 83942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1694.076706] LNet: Removed LNI 192.168.201.59@tcp [ 1694.117322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.197654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.219743] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1695.106606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1695.186504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1695.445359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.563541] LNet: Removed LNI 192.168.201.59@tcp [ 1696.626403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.755125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1696.811226] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1697.612731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1697.992694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.051254] LNet: Removed LNI 192.168.201.59@tcp [ 1699.093951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.170706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.208867] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1699.886784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1700.051570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1701.500924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.566030] LNet: Removed LNI 192.168.201.59@tcp [ 1702.614685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.725722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1702.775046] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1704.163553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1704.493495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.577146] LNet: Removed LNI 192.168.201.59@tcp [ 1705.643590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.736182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.772534] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1706.794968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1706.853707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.875761] LNet: Removed LNI 192.168.201.59@tcp [ 1706.891195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.938740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.959914] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1708.044456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1708.157307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.192871] LNet: Removed LNI 192.168.201.59@tcp [ 1708.225741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.363774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1708.525813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.596922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.698298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1708.816118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.865153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.973284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1709.106973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.146989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.198600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1709.229916] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1710.194700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.229464] LNet: 84976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.234866] LNet: 84976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1710.241857] LNetError: 84976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.246625] LNetError: 84976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1710.264783] LNet: Removed LNI 192.168.201.59@tcp [ 1710.286618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.365912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1719.655853] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:43:01 (1789429381) [ 1719.947937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.996479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.063732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1720.091900] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1721.972829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.034274] LNet: Removed LNI 192.168.201.59@tcp231 [ 1722.083838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.246107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1722.279087] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1722.282988] LNet: 85704:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.042261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1723.094649] LNet: Removed LNI 192.168.201.59@tcp231 [ 1723.140752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1723.179194] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1724.507455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.559772] LNet: Removed LNI 192.168.201.59@tcp231 [ 1724.596494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.676816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1724.709073] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1724.715741] LNet: 85856:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1725.225739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1725.284719] LNet: Removed LNI 192.168.201.59@tcp231 [ 1725.309730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1725.358472] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1727.349885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.401342] LNet: Removed LNI 192.168.201.59@tcp231 [ 1727.431148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.502713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1727.556915] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1727.563683] LNet: 86008:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1728.286595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1728.357436] LNet: Removed LNI 192.168.201.59@tcp231 [ 1728.388223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1728.471350] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1730.411525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.496861] LNet: Removed LNI 192.168.201.59@tcp231 [ 1730.541420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.702596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1730.740877] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1730.748772] LNet: 86159:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.459745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1731.536971] LNet: Removed LNI 192.168.201.59@tcp231 [ 1731.588989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1731.633656] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1742.785161] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:43:24 (1789429404) [ 1743.434504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.472468] LNet: 86870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.490811] LNet: 86870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1743.501791] LNetError: 86870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.508091] LNetError: 86870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1743.550495] LNet: Removed LNI 192.168.201.59@tcp231 [ 1743.589265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.683280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1743.732537] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1744.681801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1744.726327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1744.792760] LNet: Removed LNI 192.168.201.59@tcp232 [ 1744.862554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1744.901099] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1746.076400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1746.171630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1746.262812] LNet: Removed LNI 192.168.201.59@tcp232 [ 1746.303225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1746.354416] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1747.458487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1747.514182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1747.580537] LNet: Removed LNI 192.168.201.59@tcp232 [ 1747.606674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1747.640935] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1748.836840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1748.900428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1748.948632] LNet: Removed LNI 192.168.201.59@tcp232 [ 1748.988616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1756.025846] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:43:38 (1789429418) [ 1757.556471] Key type .llcrypt unregistered [ 1757.559837] Key type ._llcrypt unregistered [ 1764.760795] Key type ._llcrypt registered [ 1764.767354] Key type .llcrypt registered [ 1765.050991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1765.067676] alg: No test for adler32 (adler32-zlib) [ 1766.151151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1766.191126] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1775.561335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1786.598358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.636789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1786.686463] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1788.648956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1788.719425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1788.786887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1788.861072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1788.951643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1789.019372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1789.136384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1789.242252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1789.327170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1789.379571] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1789.399254] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1789.812087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1789.863683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1789.908657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1789.952492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1789.991551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1790.035420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1790.083425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1790.128490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1790.178581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1790.199630] LNet: 88660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1790.234552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1790.292372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1790.337426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1790.415432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1790.494950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1790.545438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1791.298310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1791.335878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1791.392656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1791.444596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1791.541658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1791.612405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1791.671413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1791.718471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1791.806255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1791.847932] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1792.187691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1792.234956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1792.293356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1792.327969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1792.369904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1792.412590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1792.450378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1792.496266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.558190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1792.596476] LNet: 88782:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.630163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1792.689721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1792.747061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1792.817407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1792.865335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1792.919817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1793.599822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1793.639659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1793.665774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1793.698578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1793.743681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1793.799499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1793.855267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1793.907541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1793.929271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1793.938768] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1794.107314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1794.125484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1794.158953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1794.187634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1794.226421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1794.275762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1794.309108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1794.358869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1794.407768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1794.427807] LNet: 88904:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1794.465818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1794.518587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1794.573661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1794.647841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1794.673754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1794.705215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1795.349718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1795.417934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1795.492283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1795.557926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1795.608406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1795.678367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1795.729950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1795.784877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1795.857345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1796.288749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1796.329506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1796.384531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1796.441151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1796.500682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1796.538766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1796.584850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1796.652028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1796.690968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1796.715520] LNet: 89027:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.771788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1796.827929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1796.864257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1796.928345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1796.972614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1797.066900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1797.851170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1797.911455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1797.975234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1798.034944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1798.070736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1798.122276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1798.188427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1798.253630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1798.303623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1798.342789] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1798.352324] LNet: Skipped 1 previous similar message [ 1798.659917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.685737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1798.727601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.765793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.795371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1798.828878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.857904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.889941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.928370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1798.973903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1799.017194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1799.049604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1799.125443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1799.192272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1799.231379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.782834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1799.811956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1799.843317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1799.873328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1799.908340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1799.948108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1799.991958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1800.023035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1800.063683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1800.090691] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1800.427471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1800.475716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1800.522023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1800.568141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1800.626817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1800.705620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1800.770409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1800.838497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.889896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.936173] LNet: 89270:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.959724] LNet: 89270:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1800.985039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1801.030411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1801.100384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1801.176882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1801.225872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1801.261193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.555085] LNet: 89345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.561425] LNetError: 89345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.607264] LNet: Removed LNI 192.168.201.59@tcp [ 1808.610168] LNetError: 89345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.621711] LNet: Removed LNI 192.168.201.59@tcp1 [ 1809.242890] Key type .llcrypt unregistered [ 1809.247337] Key type ._llcrypt unregistered [ 1829.133930] Key type ._llcrypt registered [ 1829.137600] Key type .llcrypt registered [ 1830.848948] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:44:52 (1789429492) [ 1831.287938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.305114] alg: No test for adler32 (adler32-zlib) [ 1832.354618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.492840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.565456] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1833.687272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1833.720815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1833.755910] LNetError: 90323:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1833.884590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.59@tcp [ 1833.970485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1841.596564] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:45:03 (1789429503) [ 1841.966870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.998139] LNet: 90913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.011438] LNetError: 90913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.025759] LNet: Removed LNI 192.168.201.59@tcp [ 1842.064182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.202553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1842.238619] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1843.243693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1843.269432] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1844.679598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.59@tcp [ 1844.701317] LNetError: 91179:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1845.215937] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1845.226011] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1845.983196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.59@tcp [ 1857.194980] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:45:18 (1789429518) [ 1859.446258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg159-server [ 1862.231257] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:45:23 (1789429523) [ 1864.909787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg159-server [ 1867.390241] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:45:29 (1789429529) [ 1869.119824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg159-server [ 1871.860900] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:45:33 (1789429533) [ 1872.140248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.178823] LNet: 92810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.186144] LNetError: 92810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.200206] LNet: Removed LNI 192.168.201.59@tcp [ 1872.215170] LNet: Removed LNI 192.168.201.59@tcp1 [ 1872.262099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.364791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.406248] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1873.057514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1873.119610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1873.179815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1873.293375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1873.328312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1873.388803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1883.985968] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:45:45 (1789429545) [ 1884.265634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.333716] LNet: 93543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.346055] LNetError: 93543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.349080] LNetError: 93543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1884.367455] LNet: Removed LNI 192.168.201.59@tcp [ 1884.388867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.455486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.508940] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1894.672402] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:45:56 (1789429556) [ 1894.928625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.021993] LNet: 94244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.035987] LNetError: 94244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.087673] LNet: Removed LNI 192.168.201.59@tcp [ 1895.140114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.239689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.285177] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1904.744617] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:46:06 (1789429566) [ 1906.562326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1908.875723] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:46:10 (1789429570) [ 1909.515844] LNet: 95283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.528842] LNetError: 95283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.545806] LNet: Removed LNI 192.168.201.59@tcp [ 1910.164867] Key type .llcrypt unregistered [ 1910.168210] Key type ._llcrypt unregistered [ 1919.216097] Key type ._llcrypt registered [ 1919.220285] Key type .llcrypt registered [ 1919.489884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1919.506989] alg: No test for adler32 (adler32-zlib) [ 1920.520884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1920.561845] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1931.184460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1937.776132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.159@tcp [ 1939.301229] LNet: 95698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.309456] LNetError: 95698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.328189] LNet: Removed LNI 192.168.201.59@tcp [ 1940.116617] Key type .llcrypt unregistered [ 1940.118757] Key type ._llcrypt unregistered [ 1955.546196] Key type ._llcrypt registered [ 1955.551286] Key type .llcrypt registered [ 1957.521060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1959.563114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1961.612375] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:47:03 (1789429623) [ 1963.412280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1965.600430] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:47:07 (1789429627) [ 1967.784841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg159-server [ 1970.304968] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:47:11 (1789429631) [ 1972.998903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg159-server [ 1974.936319] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:47:16 (1789429636) [ 1976.776766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.803084] alg: No test for adler32 (adler32-zlib) [ 1977.933820] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 1978.432543] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1980.151285] Key type lgssc registered [ 1982.104663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2009.032399] Key type lgssc unregistered [ 2009.394172] LNet: 98871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.415408] LNetError: 98871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.433895] LNet: Removed LNI 192.168.201.59@tcp [ 2010.666619] Key type .llcrypt unregistered [ 2010.672415] Key type ._llcrypt unregistered [ 2029.595498] Key type ._llcrypt registered [ 2029.599431] Key type .llcrypt registered [ 2030.869959] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:48:13 (1789429693) [ 2031.653897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.686515] alg: No test for adler32 (adler32-zlib) [ 2032.636490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.761157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2032.840528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2032.933515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2032.983517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2041.816774] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:48:23 (1789429703) [ 2042.270232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2042.319516] LNet: 100283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.358659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.446736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.489163] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2043.694634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2043.746574] LNet: 100428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.758750] LNetError: 100428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.785348] LNet: Removed LNI 192.168.201.59@tcp [ 2043.808803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2043.873258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2043.910069] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2044.659381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.688909] LNetError: 100456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.701758] LNet: Removed LNI 192.168.201.59@tcp [ 2044.725988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.815913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2044.863555] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2053.091805] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:48:34 (1789429714) [ 2053.253510] LNet: 101039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.264508] LNet: 101039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2053.274290] LNetError: 101039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.296175] LNet: Removed LNI 192.168.201.59@tcp [ 2053.956972] Key type .llcrypt unregistered [ 2053.959195] Key type ._llcrypt unregistered [ 2061.973852] Key type ._llcrypt registered [ 2061.979357] Key type .llcrypt registered [ 2062.427221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2062.460107] alg: No test for adler32 (adler32-zlib) [ 2063.430582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.540793] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2063.546964] LNet: *** cfs_fail_loc=e003, val=100*** [ 2064.682923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2064.703224] LNet: *** cfs_fail_loc=e003, val=100*** [ 2064.706511] LNet: Skipped 2 previous similar messages [ 2064.713957] Ping reply received with mapping: [ 2064.716515] found metadata reply NI[192.168.201.59@tcp] [ 2064.721732] 3b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2064.726040] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2064.729725] 5a 5a 5a 5a 5a 5a 5a 5a [ 2064.861466] LNet: 101442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.871356] LNetError: 101442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.891069] LNet: Removed LNI 192.168.201.59@tcp [ 2065.776806] Key type .llcrypt unregistered [ 2065.779061] Key type ._llcrypt unregistered [ 2080.002865] Key type ._llcrypt registered [ 2080.004670] Key type .llcrypt registered [ 2081.657082] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:49:03 (1789429743) [ 2082.450333] Key type .llcrypt unregistered [ 2082.456806] Key type ._llcrypt unregistered [ 2091.711627] Key type ._llcrypt registered [ 2091.718128] Key type .llcrypt registered [ 2091.993565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.033731] alg: No test for adler32 (adler32-zlib) [ 2093.498672] LNetError: 102479:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2093.988077] Key type .llcrypt unregistered [ 2093.990686] Key type ._llcrypt unregistered [ 2101.737207] Key type ._llcrypt registered [ 2101.739982] Key type .llcrypt registered [ 2103.426891] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:49:25 (1789429765) [ 2104.413423] Key type .llcrypt unregistered [ 2104.418718] Key type ._llcrypt unregistered [ 2112.405620] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2112.431701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2115.120231] Key type ._llcrypt registered [ 2115.130336] Key type .llcrypt registered [ 2115.321905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.339279] alg: No test for adler32 (adler32-zlib) [ 2116.353454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.466061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2128.761912] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:49:50 (1789429790) [ 2129.006384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.068149] LNet: 104251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.082752] LNetError: 104251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2129.128577] LNet: Removed LNI 10.1.2.3@tcp [ 2129.179503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.182870] Key type .llcrypt unregistered [ 2130.186555] Key type ._llcrypt unregistered [ 2140.234230] Key type ._llcrypt registered [ 2140.250154] Key type .llcrypt registered [ 2140.558621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2140.596206] alg: No test for adler32 (adler32-zlib) [ 2141.674330] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2141.821724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2142.087328] LNet: 104541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.094545] LNetError: 104541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.132921] LNet: Removed LNI 192.168.201.59@tcp [ 2143.363659] Key type .llcrypt unregistered [ 2143.372274] Key type ._llcrypt unregistered [ 2158.360452] Key type ._llcrypt registered [ 2158.362407] Key type .llcrypt registered [ 2160.199976] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:50:21 (1789429821) [ 2161.251715] Key type .llcrypt unregistered [ 2161.257818] Key type ._llcrypt unregistered [ 2169.483775] Key type ._llcrypt registered [ 2169.495942] Key type .llcrypt registered [ 2169.783664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2169.811445] alg: No test for adler32 (adler32-zlib) [ 2178.796668] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:50:40 (1789429840) [ 2178.876348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.934619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.059961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2179.099617] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2179.973386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.031955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2180.084239] LNet: Removed LNI 192.168.201.59@tcp [ 2180.310585] LNet: 106285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.750549] Key type .llcrypt unregistered [ 2180.752213] Key type ._llcrypt unregistered [ 2196.136429] Key type ._llcrypt registered [ 2196.146209] Key type .llcrypt registered [ 2197.947790] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:50:59 (1789429859) [ 2198.494954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2198.513432] alg: No test for adler32 (adler32-zlib) [ 2199.494143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.576063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.619118] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2200.755953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2200.811820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.59@tcp [ 2200.849868] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2200.855894] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2200.861461] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2200, 0, 0 [ 2200.872814] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2200.878614] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2200.885893] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2200.890129] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2200, 0, 0 [ 2200.897986] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2200.904075] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2200.911152] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2200.917983] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2200, 0, 2250 [ 2200.925955] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2200.931051] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2200.938194] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2200.943860] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2200, 0, 2250 [ 2200.953159] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2210.770846] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:51:12 (1789429872) [ 2211.611726] LNet: 107824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.620306] LNetError: 107824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.635987] LNet: Removed LNI 192.168.201.59@tcp [ 2212.923635] Key type .llcrypt unregistered [ 2212.929289] Key type ._llcrypt unregistered [ 2223.286772] Key type ._llcrypt registered [ 2223.292236] Key type .llcrypt registered [ 2223.768749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2223.803672] alg: No test for adler32 (adler32-zlib) [ 2224.908141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2224.940388] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2234.996508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2248.328937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.444069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2248.500783] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2250.139298] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2250.561674] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2250.586692] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2250.596413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2253.119637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2253.153404] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2268.459826] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:52:10 (1789429930) [ 2269.128067] LNet: 109331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.142446] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.154403] LNet: Removed LNI 192.168.201.59@tcp [ 2270.175238] LNet: Removed LNI 192.168.201.59@tcp1 [ 2270.180432] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2270.201256] LNetError: 109331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2270.245062] LNet: Removed LNI 10.1.2.3@tcp99 [ 2271.391201] Key type .llcrypt unregistered [ 2271.398821] Key type ._llcrypt unregistered [ 2281.669956] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2281.679228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2284.539743] Key type ._llcrypt registered [ 2284.542392] Key type .llcrypt registered [ 2284.794958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2284.817980] alg: No test for adler32 (adler32-zlib) [ 2285.812706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.888302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2285.918079] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2287.028240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2287.065601] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2290.891264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2291.000278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2291.077701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2291.104450] LNetError: 110073:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2291.162890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2291.237221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2291.913528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2292.080525] LNet: 110093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.115453] LNetError: 110093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2292.132961] LNet: Removed LNI 10.1.2.3@tcp [ 2293.163236] LNet: Removed LNI 192.168.201.59@tcp [ 2294.084717] Key type .llcrypt unregistered [ 2294.092832] Key type ._llcrypt unregistered [ 2312.312330] Key type ._llcrypt registered [ 2312.317069] Key type .llcrypt registered [ 2313.856116] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:52:55 (1789429975) [ 2314.452094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2314.485751] alg: No test for adler32 (adler32-zlib) [ 2315.372263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.446191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2315.470148] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2316.588904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2316.652255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg159-client@tcp [ 2324.782293] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:53:06 (1789429986) [ 2325.173867] LNet: 111625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.186987] LNetError: 111625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.212630] LNet: Removed LNI 192.168.201.59@tcp [ 2326.031140] Key type .llcrypt unregistered [ 2326.032746] Key type ._llcrypt unregistered [ 2334.172244] Key type ._llcrypt registered [ 2334.173466] Key type .llcrypt registered [ 2334.461162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2334.507391] alg: No test for adler32 (adler32-zlib) [ 2335.546695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2335.594586] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2335.629730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2343.809538] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:53:25 (1789430005) [ 2345.369428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2348.094488] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:53:29 (1789430009) [ 2350.142535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2352.350994] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:53:34 (1789430014) [ 2352.478762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.512933] LNet: 113041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.528788] LNetError: 113041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.560195] LNet: Removed LNI 192.168.201.59@tcp [ 2352.595540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.676671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2352.702873] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2353.654561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2353.719939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2361.694263] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:53:43 (1789430023) [ 2362.039774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.087303] LNet: 113756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2362.097487] LNetError: 113756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.128955] LNet: Removed LNI 192.168.201.59@tcp [ 2362.204773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.278695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2362.317233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2363.026773] Key type .llcrypt unregistered [ 2363.029190] Key type ._llcrypt unregistered [ 2379.053960] Key type ._llcrypt registered [ 2379.056889] Key type .llcrypt registered [ 2381.589346] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:54:02 (1789430042) [ 2382.005430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2382.020776] alg: No test for adler32 (adler32-zlib) [ 2383.015579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.110303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2383.143881] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2384.202333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2384.322941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2392.493717] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:54:14 (1789430054) [ 2392.690066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.729579] LNet: 115309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.737543] LNetError: 115309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.748971] LNet: Removed LNI 192.168.201.59@tcp [ 2392.769544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.835876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2392.871178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2400.886589] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:54:22 (1789430062) [ 2401.118431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.180311] LNet: 115890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.240436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.356393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2401.892547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2401.941610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2402.026536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2402.799790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2402.857578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2403.030597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.081830] LNet: 115923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.136894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.276267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2403.860048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2403.923665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2404.023381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2404.759617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2404.822250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2412.760084] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:54:34 (1789430074) [ 2413.173790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.193526] LNet: 116524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.218384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.297441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2413.339705] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2414.147212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2414.196855] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2415.394503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2415.468379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2415.773131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.786544] LNetError: 116803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.794374] LNet: Removed LNI 192.168.201.59@tcp [ 2415.807473] LNet: Removed LNI 192.168.201.59@tcp2 [ 2415.850743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.918261] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2415.930802] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2415.942441] LNet: 116809:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.731916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2417.104364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.152356] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2417.172230] LNetError: 116846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2417.183728] LNet: Removed LNI 192.168.201.59@tcp [ 2417.200439] LNet: Removed LNI 192.168.201.59@tcp2 [ 2417.221828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.331723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2417.402024] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2418.351804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2418.378191] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2419.564892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2419.612797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2419.731191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2420.031197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2420.052602] LNet: 117131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2420.061206] LNet: 117131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2420.068656] LNetError: 117131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.073682] LNetError: 117131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2420.078197] LNet: Removed LNI 192.168.201.59@tcp [ 2421.105570] LNet: Removed LNI 192.168.201.59@tcp2 [ 2421.176927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.359438] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2421.369269] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2421.376236] LNet: 117137:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.330199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2422.587651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2422.617463] LNet: Removed LNI 192.168.201.59@tcp [ 2423.653719] LNet: Removed LNI 192.168.201.59@tcp2 [ 2423.729199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.790714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2423.814695] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2424.725114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2424.747376] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2425.751702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2425.814214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2426.051851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.076115] LNetError: 117454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.083638] LNetError: 117454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2426.090766] LNet: Removed LNI 192.168.201.59@tcp [ 2426.105287] LNet: Removed LNI 192.168.201.59@tcp2 [ 2426.127134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.256729] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2426.272558] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2426.275987] LNet: 117460:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2427.010199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2427.297995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2428.383302] LNet: Removed LNI 192.168.201.59@tcp [ 2428.395380] LNet: Removed LNI 192.168.201.59@tcp2 [ 2428.454335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2428.519703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2428.556483] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2429.692315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2429.708766] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2430.653923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2430.707845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2430.817280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2431.186679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.208930] LNet: 117781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.216646] LNet: 117781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2432.287354] LNet: Removed LNI 192.168.201.59@tcp [ 2433.320038] LNet: Removed LNI 192.168.201.59@tcp2 [ 2433.379538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.466619] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2433.475323] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2433.484481] LNet: 117788:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.317538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2434.658938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2434.736341] LNetError: 117825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.745618] LNetError: 117825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2434.753828] LNet: Removed LNI 192.168.201.59@tcp [ 2435.835577] LNet: Removed LNI 192.168.201.59@tcp2 [ 2435.880870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.970977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2436.014422] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2436.956717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2436.977977] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2437.766327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2437.813766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2437.895550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2438.312517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2439.391682] LNet: Removed LNI 192.168.201.59@tcp [ 2439.411900] LNet: Removed LNI 192.168.201.59@tcp2 [ 2439.456044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2439.539042] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2439.551286] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2439.556643] LNet: 118128:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2440.042727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2440.466861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.569118] LNet: Removed LNI 192.168.201.59@tcp [ 2441.589775] LNet: Removed LNI 192.168.201.59@tcp2 [ 2441.640995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.751195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2441.789779] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2442.732963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2442.763695] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2443.761889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2443.804632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2443.850992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2443.971608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2444.451954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.535835] LNet: Removed LNI 192.168.201.59@tcp [ 2446.566991] LNet: Removed LNI 192.168.201.59@tcp2 [ 2446.619558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.773231] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2446.779951] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2447.756770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2457.143937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2459.324505] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:55:21 (1789430121) [ 2459.645389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2459.668916] LNet: 119248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.674386] LNet: 119248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2459.683324] LNetError: 119248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2459.690219] LNetError: 119248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2459.703625] LNet: Removed LNI 192.168.201.59@tcp [ 2460.778400] LNet: Removed LNI 192.168.201.59@tcp2 [ 2460.828302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2461.390879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2461.409176] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2462.300962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2462.320338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2463.262229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2463.280465] LNet: Removed LNI 10.1.2.3@tcp [ 2463.293231] LNet: Removed LNI 192.168.201.59@tcp [ 2463.335497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.406409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2463.428511] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2463.434976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2463.996972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2464.031919] LNet: Removed LNI 192.168.201.59@tcp [ 2464.053523] LNet: Removed LNI 10.1.2.3@tcp [ 2471.758995] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:55:33 (1789430133) [ 2471.911938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.988985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.091968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2472.144378] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2472.935465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2472.989277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2473.044048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.086580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.127721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.160245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.201325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.263891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.342816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.410319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.459661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.533726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2473.835417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2480.902790] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 19:55:42 (1789430142) [ 2481.029862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2482.087069] LNet: Removed LNI 192.168.201.59@tcp [ 2482.150920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.247831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2482.300538] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2483.118624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2483.180218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2483.255895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.323567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.369614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.437755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.488510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.555178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.599715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.663787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.701985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.735899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.870965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2483.910343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.943798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2483.989630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.028848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.060266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.088616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.118913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.154535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.183612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2484.209607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2490.901266] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:55:53 (1789430153) [ 2491.058872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2492.136609] LNet: Removed LNI 192.168.201.59@tcp [ 2492.203150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.439654] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2492.681413] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2492.685172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2492.691538] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2495.130239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2495.174399] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2496.439560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2496.472497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2499.705956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2499.742315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2499.795567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.59@tcp -m PUT -e local_timeout -r 1 [ 2499.847287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2499.892113] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2499.897044] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2500.148829] LNetError: 121686:0:(lib-move.c:4794:lnet_parse()) 192.168.201.59@tcp, src 192.168.201.59@tcp: Dropping PUT (error -108 looking up sender) [ 2501.151813] LNet: 122144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2501.169780] LNet: 122144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2501.181717] LNetError: 122144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2501.203657] LNetError: 122144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2501.228370] LNet: Removed LNI 10.1.2.3@tcp [ 2502.253992] LNet: Removed LNI 192.168.201.59@tcp [ 2502.809565] Key type .llcrypt unregistered [ 2502.813560] Key type ._llcrypt unregistered [ 2516.754822] Key type ._llcrypt registered [ 2516.756204] Key type .llcrypt registered [ 2517.810572] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:56:20 (1789430180) [ 2518.210277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.223830] alg: No test for adler32 (adler32-zlib) [ 2519.201377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2519.556045] LNet: 122994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.260542] Key type .llcrypt unregistered [ 2520.263989] Key type ._llcrypt unregistered [ 2527.561448] Key type ._llcrypt registered [ 2527.563398] Key type .llcrypt registered [ 2527.853585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2527.883526] alg: No test for adler32 (adler32-zlib) [ 2528.881728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2528.923922] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2537.625221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2537.799250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2538.375513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2538.686686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2538.715855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2538.755983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2538.798778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2538.839888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2538.880757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2538.933957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2538.989882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2539.059230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2539.075593] LNet: 123443:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2539.098694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2539.174432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2539.181827] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2539.218838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2539.284934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2539.356699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2539.429761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2540.249162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2540.564506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2540.604728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2540.646872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2540.678959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2540.716981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2540.744135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2540.780447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2540.828507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2540.879849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2540.917176] LNet: 123562:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2540.954844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2540.998960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2541.056763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2541.114612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2541.156800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2541.217728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.139781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2542.531642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2542.563973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2542.604352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2542.660630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2542.692548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2542.723750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2542.756519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2542.782706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2542.814949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2542.831571] LNet: 123681:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2542.860590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2542.936910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.002291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.049551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.085445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.144722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2544.075792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2544.486386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.547890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2544.596686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.643669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.705683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2544.754944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.791232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.819831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.846380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.863874] LNet: 123800:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2544.880795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.911669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.950086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.996885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2545.035448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2545.072617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2545.423362] LNet: 123846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2545.429207] LNetError: 123846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.447146] LNet: Removed LNI 192.168.201.59@tcp [ 2545.896606] Key type .llcrypt unregistered [ 2545.901061] Key type ._llcrypt unregistered [ 2566.952953] Key type ._llcrypt registered [ 2566.955474] Key type .llcrypt registered [ 2568.724617] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:57:10 (1789430230) [ 2569.290913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.309381] alg: No test for adler32 (adler32-zlib) [ 2570.431442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2571.031179] LNet: 124732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.811672] Key type .llcrypt unregistered [ 2571.815866] Key type ._llcrypt unregistered [ 2580.386171] Key type ._llcrypt registered [ 2580.388263] Key type .llcrypt registered [ 2580.633865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2580.652161] alg: No test for adler32 (adler32-zlib) [ 2581.642535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2581.705722] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2591.866604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2605.082317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2605.184758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2605.214109] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2606.851643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2607.741451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2608.097040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2608.130036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2608.165100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2608.195993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2608.227700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2608.276685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2608.327425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2608.354156] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2608.400728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2608.489657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2608.526793] LNet: 125385:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2608.570640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2608.625668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2608.687371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2608.760580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2608.822939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2608.895344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2610.049517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2610.452600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2610.503345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2610.547869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2610.606737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2610.650219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2610.697753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2610.770254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2610.791680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2610.822595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2610.841672] LNet: 125504:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2610.869223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2610.904586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2610.961851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2611.021389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2611.061189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2611.116366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2611.940263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2612.228789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2612.264442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2612.307471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2612.361687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2612.411171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2612.461625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2612.519576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2612.572204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2612.625628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2612.669125] LNet: 125623:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2612.713298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2612.789479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2612.910869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2612.986511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2613.030483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2613.086474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2613.609467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2613.841104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2613.884892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2613.912187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2613.942100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2613.969651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2613.996860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2614.029596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2614.087747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2614.141620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2614.182869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2614.259781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2614.314453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2614.359543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2614.424886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2614.488656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2620.724137] LNet: 125823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.731230] LNetError: 125823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.738048] LNet: Removed LNI 192.168.201.59@tcp [ 2620.752184] LNet: Removed LNI 192.168.201.59@tcp1 [ 2621.423931] Key type .llcrypt unregistered [ 2621.430851] Key type ._llcrypt unregistered [ 2647.815956] Key type ._llcrypt registered [ 2647.820276] Key type .llcrypt registered [ 2649.385783] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:58:31 (1789430311) [ 2649.999161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.027193] alg: No test for adler32 (adler32-zlib) [ 2651.148829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2651.239284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2651.304632] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2652.489960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2652.524045] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2653.438253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2653.522173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2654.559121] LNet: Removed LNI 192.168.201.59@tcp [ 2663.137754] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:58:44 (1789430324) [ 2665.828093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2668.768687] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:58:49 (1789430329) [ 2680.868346] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:59:02 (1789430342) [ 2691.319499] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:59:12 (1789430352) [ 2703.993323] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:59:25 (1789430365) [ 2713.773158] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:59:35 (1789430375) [ 2722.885531] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:59:44 (1789430384) [ 2733.366300] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:59:55 (1789430395) [ 2742.446833] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:00:04 (1789430404) [ 2749.575738] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:00:11 (1789430411) [ 2758.894448] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:00:21 (1789430421) [ 2767.760443] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:00:29 (1789430429) [ 2776.732816] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:00:39 (1789430439) [ 2785.588996] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:00:47 (1789430447) [ 2795.358240] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:00:57 (1789430457) [ 2803.821578] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:01:06 (1789430466) [ 2811.066155] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:01:13 (1789430473) [ 2818.741466] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:01:20 (1789430480) [ 2826.443190] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:01:28 (1789430488) [ 2826.658868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2826.676648] LNet: 137534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2826.681978] LNetError: 137534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2827.756097] LNet: Removed LNI 192.168.201.59@tcp2 [ 2827.803083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2827.977472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2828.010525] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2829.613605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2829.655588] LNet: 137676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.665550] LNetError: 137676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.680904] LNet: Removed LNI 192.168.201.59@tcp [ 2829.706855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2829.761640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2829.786852] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2837.280236] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:01:39 (1789430499) [ 2846.447782] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:01:48 (1789430508) [ 2855.666236] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:01:57 (1789430517) [ 2862.815394] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:02:05 (1789430525) [ 2870.431987] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:02:12 (1789430532) [ 2879.056434] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:02:20 (1789430540) [ 2887.497795] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:02:29 (1789430549) [ 2895.200491] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:02:37 (1789430557) [ 2903.403622] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:02:45 (1789430565) [ 2911.653444] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:02:53 (1789430573) [ 2919.472314] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:03:01 (1789430581) [ 2926.700451] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:03:08 (1789430588) [ 2936.708791] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2684 sec ========== 20:03:18 (1789430598) [ 2937.344435] LNet: 145144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.349032] LNetError: 145144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.378278] LNet: Removed LNI 192.168.201.59@tcp [ 2938.627354] Key type .llcrypt unregistered [ 2938.630523] Key type ._llcrypt unregistered [ 2955.852655] Key type ._llcrypt registered [ 2955.855261] Key type .llcrypt registered [ 2956.430985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2956.439545] alg: No test for adler32 (adler32-zlib) [ 2957.951491] Lustre: Lustre: Build Version: 2.17.58_41_g9380d02 [ 2958.334504] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2960.151257] Key type lgssc registered [ 2961.840784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3032.361642] Lustre: Mounted lustre-client - version 2.17.58_41_g9380d02 [ 3037.837477] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3046.058458] Lustre: Unmounted lustre-client [ 3083.440993] Key type lgssc unregistered [ 3083.800389] LNet: 148094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3083.804934] LNetError: 148094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3083.833441] LNet: Removed LNI 192.168.201.59@tcp [ 3084.590188] Key type .llcrypt unregistered [ 3084.595340] Key type ._llcrypt unregistered