[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478402489 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2560MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002367] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007517] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008028] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.010159] LSM: Security Framework initializing [ 0.011072] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013066] *** VALIDATE selinux *** [ 0.021771] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027353] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029023] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031157] *** VALIDATE tmpfs *** [ 0.032516] *** VALIDATE proc *** [ 0.034154] *** VALIDATE cgroup *** [ 0.035006] *** VALIDATE cgroup2 *** [ 0.036243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037148] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044307] debug: unmapping init [mem 0xffffffffa4a59000-0xffffffffa4a60fff] [ 0.046292] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047710] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048031] ... version: 2 [ 0.049011] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051008] ... value mask: 0000ffffffffffff [ 0.052009] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054012] ... event mask: 000000070000000f [ 0.056288] rcu: Hierarchical SRCU implementation. [ 0.058915] smp: Bringing up secondary CPUs ... [ 0.059556] x86: Booting SMP configuration: [ 0.060019] .... node #0, CPUs: #1 #2 #3 [ 0.098129] smp: Brought up 1 node, 4 CPUs [ 0.100035] smpboot: Max logical packages: 1 [ 0.101033] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.149024] node 0 deferred pages initialised in 44ms [ 0.151093] devtmpfs: initialized [ 0.153515] x86/mm: Memory block size: 128MB [ 0.157935] gcov: version magic: 0x41383552 [ 0.161274] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163259] pinctrl core: initialized pinctrl subsystem [ 0.164170] [ 0.165008] ************************************************************* [ 0.166176] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167009] ** ** [ 0.168011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169026] ** ** [ 0.170012] ** This means that this kernel is built to expose internal ** [ 0.171012] ** IOMMU data structures, which may compromise security on ** [ 0.172032] ** your system. ** [ 0.173016] ** ** [ 0.174011] ** If you see this message and you are not debugging the ** [ 0.175012] ** kernel, report this immediately to your vendor! ** [ 0.176015] ** ** [ 0.177012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178012] ************************************************************* [ 0.180000] NET: Registered protocol family 16 [ 0.182718] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197058] cpuidle: using governor menu [ 0.199028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202048] PCI: Using configuration type 1 for base access [ 0.204136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214131] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.217020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.221128] cryptd: max_cpu_qlen set to 1000 [ 0.223261] ACPI: Added _OSI(Module Device) [ 0.225014] ACPI: Added _OSI(Processor Device) [ 0.227020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.229015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.243430] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247000] ACPI: Interpreter enabled [ 0.250165] ACPI: PM: (supports S0 S3 S4 S5) [ 0.253013] ACPI: Using IOAPIC for interrupt routing [ 0.257165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281718] acpiphp: Slot [2] registered [ 0.283136] acpiphp: Slot [5] registered [ 0.290252] acpiphp: Slot [6] registered [ 0.291141] acpiphp: Slot [3] registered [ 0.292527] acpiphp: Slot [4] registered [ 0.294116] acpiphp: Slot [7] registered [ 0.360301] acpiphp: Slot [8] registered [ 0.362136] acpiphp: Slot [9] registered [ 0.364091] acpiphp: Slot [10] registered [ 0.366314] acpiphp: Slot [11] registered [ 0.367250] acpiphp: Slot [12] registered [ 0.369091] acpiphp: Slot [13] registered [ 0.370237] acpiphp: Slot [14] registered [ 0.371116] acpiphp: Slot [15] registered [ 0.373185] acpiphp: Slot [16] registered [ 0.374098] acpiphp: Slot [17] registered [ 0.376110] acpiphp: Slot [18] registered [ 0.377068] acpiphp: Slot [19] registered [ 0.378156] acpiphp: Slot [20] registered [ 0.380091] acpiphp: Slot [21] registered [ 0.382098] acpiphp: Slot [22] registered [ 0.384125] acpiphp: Slot [23] registered [ 0.385099] acpiphp: Slot [24] registered [ 0.388140] acpiphp: Slot [25] registered [ 0.389082] acpiphp: Slot [26] registered [ 0.391147] acpiphp: Slot [27] registered [ 0.393137] acpiphp: Slot [28] registered [ 0.394091] acpiphp: Slot [29] registered [ 0.396125] acpiphp: Slot [30] registered [ 0.398108] acpiphp: Slot [31] registered [ 0.399084] PCI host bridge to bus 0000:00 [ 0.400015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.408028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.411033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.414207] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.418086] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.423256] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.434062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.437248] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.440023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.445906] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.449006] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.452047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.456070] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.460762] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.470000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.471000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.478034] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.485023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.494020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.509020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.522712] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.529018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.534021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.538155] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.541214] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.542000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.542305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.543000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.552149] iommu: Default domain type: Passthrough [ 0.555345] SCSI subsystem initialized [ 0.558407] ACPI: bus type USB registered [ 0.561213] usbcore: registered new interface driver usbfs [ 0.578056] usbcore: registered new interface driver hub [ 0.595010] usbcore: registered new device driver usb [ 0.606000] pps_core: LinuxPPS API ver. 1 registered [ 0.609025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.619894] PTP clock support registered [ 0.624171] EDAC MC: Ver: 3.0.0 [ 0.628867] PCI: Using ACPI for IRQ routing [ 0.631000] NetLabel: Initializing [ 0.632014] NetLabel: domain hash size = 128 [ 0.637037] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.645100] NetLabel: unlabeled traffic allowed by default [ 0.650310] vgaarb: loaded [ 0.653732] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.656018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.663551] clocksource: Switched to clocksource kvm-clock [ 0.884623] VFS: Disk quotas dquot_6.6.0 [ 0.886722] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889464] *** VALIDATE ramfs *** [ 0.890722] *** VALIDATE hugetlbfs *** [ 0.893848] pnp: PnP ACPI init [ 0.896567] pnp: PnP ACPI: found 6 devices [ 0.920259] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.929767] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.945308] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.953723] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.961323] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.967715] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.975312] NET: Registered protocol family 2 [ 0.980506] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.992324] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.998678] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.009726] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015562] TCP: Hash tables configured (established 65536 bind 65536) [ 1.020702] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.025378] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.029574] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033467] NET: Registered protocol family 1 [ 1.038672] RPC: Registered named UNIX socket transport module. [ 1.040698] RPC: Registered udp transport module. [ 1.049166] RPC: Registered tcp transport module. [ 1.050801] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.052997] NET: Registered protocol family 44 [ 1.054391] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.057015] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.060143] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.063699] PCI: CLS 0 bytes, default 64 [ 1.065248] Unpacking initramfs... [ 2.803728] debug: unmapping init [mem 0xffff8c5cfcc64000-0xffff8c5cfffcffff] [ 2.814753] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.822474] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 2.831914] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.347979] Initialise system trusted keyrings [ 4.350568] Key type blacklist registered [ 4.352240] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.360974] zbud: loaded [ 4.363787] *** VALIDATE nfs *** [ 4.364742] *** VALIDATE nfs4 *** [ 4.366181] pstore: using deflate compression [ 4.369470] Platform Keyring initialized [ 4.477511] NET: Registered protocol family 38 [ 4.480661] Key type asymmetric registered [ 4.494027] Asymmetric key parser 'x509' registered [ 4.504530] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.524451] io scheduler mq-deadline registered [ 4.536613] io scheduler kyber registered [ 4.538395] io scheduler bfq registered [ 4.540766] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.543965] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.547231] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.550189] ACPI: Power Button [PWRF] [ 4.555809] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.577753] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.629316] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.690029] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.728021] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.732860] Non-volatile memory driver v1.3 [ 4.734367] Linux agpgart interface v0.103 [ 4.768427] virtio_blk virtio1: [vda] 134776 512-byte logical blocks (69.0 MB/65.8 MiB) [ 4.770925] vda: detected capacity change from 0 to 69005312 [ 4.786543] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.789499] vdb: detected capacity change from 0 to 1073741824 [ 4.801887] libphy: Fixed MDIO Bus: probed [ 4.829188] usbcore: registered new interface driver usbserial_generic [ 4.834115] usbserial: USB Serial support registered for generic [ 4.837910] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.846716] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.848565] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.851770] mousedev: PS/2 mouse device common for all mice [ 4.857687] rtc_cmos 00:05: RTC can wake from S4 [ 4.860374] rtc_cmos 00:05: registered as rtc0 [ 4.862045] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.870058] intel_pstate: CPU model not supported [ 4.871191] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.875549] hid: raw HID events driver (C) Jiri Kosina [ 4.890858] usbcore: registered new interface driver usbhid [ 4.892576] usbhid: USB HID core driver [ 4.894190] drop_monitor: Initializing network drop monitor service [ 4.896305] Initializing XFRM netlink socket [ 4.898167] NET: Registered protocol family 10 [ 4.900779] Segment Routing with IPv6 [ 4.902043] NET: Registered protocol family 17 [ 4.904296] mpls_gso: MPLS GSO support [ 4.910299] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.914161] RAS: Correctable Errors collector initialized. [ 4.916202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.924711] AVX version of gcm_enc/dec engaged. [ 4.924719] AES CTR mode by8 optimization enabled [ 5.035518] sched_clock: Marking stable (5035491089, 0)->(6318748994, -1283257905) [ 5.039366] registered taskstats version 1 [ 5.041519] Loading compiled-in X.509 certificates [ 5.043659] zswap: loaded using pool lzo/zbud [ 5.083471] Key type big_key registered [ 5.096558] Key type encrypted registered [ 5.097899] ima: No TPM chip found, activating TPM-bypass! [ 5.099752] ima: Allocated hash algorithm: sha1 [ 5.101413] ima: No architecture policies found [ 5.103360] evm: Initialising EVM extended attributes: [ 5.104826] evm: security.selinux [ 5.106406] evm: security.ima [ 5.107544] evm: security.capability [ 5.109283] evm: HMAC attrs: 0x1 [ 5.124222] rtc_cmos 00:05: setting system clock to 2026-05-15 09:55:57 UTC (1778838957) [ 5.130730] debug: unmapping init [mem 0xffffffffa5a03000-0xffffffffa5bfffff] [ 5.135317] debug: unmapping init [mem 0xffffffffa4782000-0xffffffffa4a58fff] [ 5.144102] Write protecting the kernel read-only data: 28672k [ 5.153116] debug: unmapping init [mem 0xffffffffa2e03000-0xffffffffa2ffffff] [ 5.206668] debug: unmapping init [mem 0xffffffffa3714000-0xffffffffa37fffff] [ 5.252611] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.261305] systemd[1]: Detected virtualization kvm. [ 5.262824] systemd[1]: Detected architecture x86-64. [ 5.264765] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.509779] systemd[1]: No hostname configured. [ 5.515275] systemd[1]: Set hostname to . [ 5.520582] random: systemd: uninitialized urandom read (16 bytes read) [ 5.523714] systemd[1]: Initializing machine ID from random generator. [ 5.833913] random: ln: uninitialized urandom read (6 bytes read) [ 6.428690] random: systemd: uninitialized urandom read (16 bytes read) [ 6.431978] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.440553] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 6.447767] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.530617] device-mapper: uevent: version 1.0.3 [ 8.533815] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK [ 12.557882] random: fast init done ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.016901] virtio_net virtio0 ens2: renamed from eth0 [ 13.571212] scsi host0: ata_piix [ 13.623205] scsi host1: ata_piix [ 13.627023] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 13.630677] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 19.972212] random: crng init done [ 19.974857] random: 7 urandom warning(s) missed due to ratelimiting [ 24.055179] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 27.720104] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.316537] printk: systemd: 26 output lines suppressed due to ratelimiting [ 35.508963] SELinux: Disabled at runtime. [ 35.875111] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 35.920266] systemd[1]: Detected virtualization kvm. [ 35.922633] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 39.111044] hrtimer: interrupt took 2398694 ns [ 39.495925] systemd[1]: initrd-switch-root.service: Succeeded. [ 39.508819] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 39.565897] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 39.595609] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 39.598570] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 39.691039] systemd[1]: Starting Journal Service... Starting Journal Service... [ 39.729533] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] [ 40.554344] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ 40.895951] systemd[1]: Starting udev Coldplug all Devices... Starting udev Coldplug all Devices... [ 40.950731] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 41.002396] systemd[1]: Stopped target Initrd Root File System. [ OK ] Stopped target Initrd Root File System. [ 41.028258] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 41.155158] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 41.197446] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ 41.340766] systemd[1]: sysroot-var-lib-nfs-rpc_pipefs.mount: Succeeded. [ 41.356295] systemd[1]: sysroot.mount: Succeeded. [ 41.429520] systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 43.294667] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 45.580125] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 45.718704] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 47.079267] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 47.290843] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (10s / no limit) [** ] A start job is running for Configur…only root support (11s / no limit) [*** ] A start job is running for Configur…only root support (11s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ *** ] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ *] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (15s / no limit)[ 54.886667] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (16s / no limit) [*** ] A start job is running for Configur…only root support (16s / no limit) [** ] A start job is running for Configur…only root support (17s / no limit)[ 56.536820] NFS: Registering the id_resolver key type [ 56.545839] Key type id_resolver registered [ 56.547463] Key type id_legacy registered [* ] A start job is running for Configur…only root support (17s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg155-client login: [ 158.191887] libcfs: loading out-of-tree module taints kernel. [ 158.364234] Key type ._llcrypt registered [ 158.366958] Key type .llcrypt registered [ 158.693105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 158.706173] alg: No test for adler32 (adler32-zlib) [ 160.119734] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 160.973655] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 162.779752] Key type lgssc registered [ 164.457224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 289.994586] Lustre: Mounted lustre-client [ 294.632356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 306.167697] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing check_logdir /tmp/testlogs/ [ 312.013803] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing yml_node [ 315.879776] Lustre: lustre-OST0000-osc-ffff8c5d52068000: disconnect after 24s idle [ 316.946594] Lustre: DEBUG MARKER: Client: 2.17.53.3 [ 319.447700] Lustre: DEBUG MARKER: MDS: 2.17.53.3 [ 322.773715] Lustre: DEBUG MARKER: OSS: 2.17.53.3 [ 325.338712] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri May 15 06:01:15 EDT 2026 [ 343.877310] LustreError: 7941:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c5d52068000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 343.954506] Lustre: Unmounted lustre-client [ 374.147144] Key type lgssc unregistered [ 374.503954] LNet: 8388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.520145] LNetError: 8388:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.540119] LNet: Removed LNI 192.168.201.55@tcp [ 375.478311] Key type .llcrypt unregistered [ 375.487197] Key type ._llcrypt unregistered [ 405.100882] Key type ._llcrypt registered [ 405.103611] Key type .llcrypt registered [ 405.345607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 405.366255] alg: No test for adler32 (adler32-zlib) [ 406.398599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 406.506589] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 406.553428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 411.103717] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 411.352119] LNet: 9271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.370734] LNetError: 9271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 411.402287] LNet: Removed LNI 192.168.201.55@tcp [ 412.114466] Key type .llcrypt unregistered [ 412.118579] Key type ._llcrypt unregistered [ 419.835960] Key type ._llcrypt registered [ 419.837578] Key type .llcrypt registered [ 421.352294] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:02:51 (1778839371) [ 421.793072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 421.812453] alg: No test for adler32 (adler32-zlib) [ 422.766730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.862528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.884492] LNet: 9683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.908499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.977542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.990420] LNet: 9690:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 430.306211] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:03:01 (1778839381) [ 430.529235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.562869] LNet: 10270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.604996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.724918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 437.779813] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:03:08 (1778839388) [ 437.863521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.899185] LNet: 10849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.925942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.048943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 445.636166] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:03:16 (1778839396) [ 445.960835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.977845] LNet: 11441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.005692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.099841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 452.548984] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:03:23 (1778839403) [ 452.728077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.761213] LNet: 12023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.806886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.890488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 453.266423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 461.039609] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:03:31 (1778839411) [ 461.158759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.200251] LNet: 12610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.236775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.386450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 462.239544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 462.892086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 470.735594] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:03:41 (1778839421) [ 470.927435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.011055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.094277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 478.255270] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:03:49 (1778839429) [ 478.344021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.396168] LNet: 13790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.413787] LNet: 13790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 478.496290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.667949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 478.717346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 479.176904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 479.233234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 479.873638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 479.944473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 480.561293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 480.607691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 481.350758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 481.414565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 482.084485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 482.136712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 482.734956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 482.774696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 483.363645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 483.464877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 484.057770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 484.121720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 484.735956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 484.786291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 485.341058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 485.410404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 485.986737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 486.029172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 486.604472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 486.649269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 493.745871] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:04:04 (1778839444) [ 493.918663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.977417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.042312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 494.093614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 502.139595] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:04:12 (1778839452) [ 502.562132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.639633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.796703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 502.849908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 509.415592] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:04:20 (1778839460) [ 509.504675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.567464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.695917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 509.740858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 517.546868] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:04:28 (1778839468) [ 517.668533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.693369] LNet: 16230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.696483] LNet: 16230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 517.750308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.898787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 517.930863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 525.085559] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:04:35 (1778839475) [ 525.224125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.279180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.398550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 525.448208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 532.886543] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:04:43 (1778839483) [ 533.008391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.082487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.201910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 533.232822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 541.376991] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:04:51 (1778839491) [ 541.529751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.581058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.724397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.159904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.645728] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:05:01 (1778839501) [ 550.827926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.923627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.169525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 551.436239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 559.413318] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:05:10 (1778839510) [ 559.572426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.630597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.788795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.030551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 568.256734] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:05:18 (1778839518) [ 568.360660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.403854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.936346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.278744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 576.991691] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:05:27 (1778839527) [ 577.071527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.135156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.210803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 577.281671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 584.230216] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:05:35 (1778839535) [ 584.312345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.351101] LNet: 21067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.359728] LNet: 21067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 584.381415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.451361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.523419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 592.549986] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:05:43 (1778839543) [ 592.694448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.746025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.878880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 600.278296] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:05:51 (1778839551) [ 600.354699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.413815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.503688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 606.711921] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:05:57 (1778839557) [ 606.768922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.821235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.909452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 607.351959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 614.299284] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:06:05 (1778839565) [ 614.420902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.481208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.604493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 614.627825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 621.223831] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:06:12 (1778839572) [ 621.453787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.553866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.682272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 621.743657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 629.340450] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:06:19 (1778839579) [ 629.499052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.560473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.730552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 629.785575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 636.714578] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:06:28 (1778839588) [ 636.947498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.029304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.151095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 637.203439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 644.835462] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:06:35 (1778839595) [ 644.988499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.081469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.166160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 645.221536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 645.235547] LNetError: 25799:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 645.269268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 645.288615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 652.838479] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:06:43 (1778839603) [ 653.144396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.198911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.351924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pk4qJB4mxd [ 653.439353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pk4qJB4mxd [ 653.520146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.pk4qJB4mxd [ 659.578405] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:06:50 (1778839610) [ 659.753641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.826729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.945960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 659.980433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 660.039601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 667.729887] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:06:57 (1778839617) [ 668.022330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.104939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.235073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 671.951078] LNet: Message forwarding will be enabled in 2s [ 675.040108] LNet: Message forwarding enabled [ 675.663477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 675.697616] LNet: Message forwarding will stop in 2s to 6s [ 679.257153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 680.997214] LNet: Message forwarding disabled [ 681.960871] LNet: Message forwarding will be enabled in 2s [ 685.102383] LNet: Message forwarding enabled [ 688.040681] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:07:19 (1778839639) [ 688.218947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.390538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.464952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 695.644593] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:07:26 (1778839646) [ 695.835675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.865192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.911787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 695.970174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 696.007335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 696.043114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 696.067877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 696.108022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 696.130250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 696.161247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 696.187731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 696.210573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 696.233519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 696.256223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 696.281687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 696.309497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 696.335927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 696.361637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 696.389282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 696.425020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 696.449443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 696.468536] LNetError: 28860:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 702.222382] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:07:33 (1778839653) [ 702.366327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.392637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.455819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.948858] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:07:40 (1778839660) [ 711.058724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg155-server [ 712.689793] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:07:43 (1778839663) [ 712.858487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.897024] LNet: 30359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.907533] LNet: 30359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 712.961099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.034536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.052060] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 714.724756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[56-60]@tcp [ 714.748523] LNet: 30576:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 714.840467] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.56@tcp is being used as a gateway but routing feature is not turned on [ 714.861084] LNetError: 30365:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.56@tcp has gone from up to down [ 715.819524] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.59@tcp is being used as a gateway but routing feature is not turned on [ 715.827330] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 716.856052] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 716.874215] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 718.887936] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.59@tcp is being used as a gateway but routing feature is not turned on [ 718.900028] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 721.878784] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:07:52 (1778839672) [ 722.992688] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.58@tcp is being used as a gateway but routing feature is not turned on [ 723.016677] LNetError: 30365:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 724.167309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg155-server [ 725.823820] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:07:56 (1778839676) [ 726.022611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.059197] LNetError: 30374:0:(lib-move.c:4710:lnet_parse()) 192.168.201.58@tcp, src 192.168.201.58@tcp: Dropping REPLY (error -108 looking up sender) [ 726.075750] LNetError: 30374:0:(lib-move.c:4710:lnet_parse()) Skipped 4 previous similar messages [ 726.077567] LNetError: 31509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.103049] LNet: Removed LNI 192.168.201.55@tcp [ 726.121769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.174594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.215746] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 727.114198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[56-61/2]@tcp [ 727.157792] LNet: 31627:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 727.165058] LNet: 31627:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 727.213086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[56-61/2]@tcp [ 727.235935] LNetError: 31514:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.60@tcp has gone from up to down [ 727.262548] LNetError: 31514:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 733.526700] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:08:04 (1778839684) [ 733.823735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.847658] LNetError: 32221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.894913] LNet: Removed LNI 192.168.201.55@tcp [ 734.942775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.041945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 735.111269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.147546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.192439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.237533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.266197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.308576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 735.382435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.411923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.465424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.596833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.639632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.689766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 735.771429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.839331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.936698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.008381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.045909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.103670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 736.199868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.278995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.375313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.490968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.580075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.667504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 736.736351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.801653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.894427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.005902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.063569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.122655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 737.210894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.243983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.319747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 743.297557] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:08:14 (1778839694) [ 745.194911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg155-server [ 747.274163] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:08:17 (1778839697) [ 749.663284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg155-server [ 751.585731] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:08:22 (1778839702) [ 752.812442] Key type .llcrypt unregistered [ 752.814760] Key type ._llcrypt unregistered [ 759.594885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 759.825124] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 759.848386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 759.862415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 762.760731] Key type ._llcrypt registered [ 762.765152] Key type .llcrypt registered [ 763.014413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.027266] alg: No test for adler32 (adler32-zlib) [ 763.990902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.076813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.113952] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 765.048319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 765.078147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.396281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 767.414811] LNet: Removed LNI 10.1.2.3@tcp [ 767.527939] LNet: 34344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.533485] LNetError: 34344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.542808] LNet: Removed LNI 192.168.201.55@tcp [ 768.118580] Key type .llcrypt unregistered [ 768.121089] Key type ._llcrypt unregistered [ 779.771984] Key type ._llcrypt registered [ 779.774446] Key type .llcrypt registered [ 780.993727] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:08:51 (1778839731) [ 781.613730] Key type .llcrypt unregistered [ 781.618071] Key type ._llcrypt unregistered [ 787.362743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.507846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.520676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.531419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.153314] Key type ._llcrypt registered [ 790.155173] Key type .llcrypt registered [ 790.286619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.297744] alg: No test for adler32 (adler32-zlib) [ 791.213793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.314957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.343139] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 792.261586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.281587] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.346805] LNet: 35836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.355516] LNetError: 35836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.361676] LNet: Removed LNI 192.168.201.55@tcp [ 795.373515] LNet: Removed LNI 10.1.2.3@tcp [ 795.822734] Key type .llcrypt unregistered [ 795.824258] Key type ._llcrypt unregistered [ 807.350358] Key type ._llcrypt registered [ 807.352547] Key type .llcrypt registered [ 808.710843] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:09:19 (1778839759) [ 809.162393] Key type .llcrypt unregistered [ 809.166219] Key type ._llcrypt unregistered [ 815.605455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.770619] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.775121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.782096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 818.338372] Key type ._llcrypt registered [ 818.339964] Key type .llcrypt registered [ 818.584198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.628175] alg: No test for adler32 (adler32-zlib) [ 819.597330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.674129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 819.731742] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 821.155679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 821.171427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.862480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 822.889320] LNet: Removed LNI 10.1.2.3@tcp [ 822.905107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 822.940511] LNet: Removed LNI 10.1.2.31@tcp [ 822.963427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.990896] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.543218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 823.568363] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 824.674422] LNet: 37757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.678461] LNetError: 37757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.682385] LNet: Removed LNI 10.1.2.3@tcp [ 824.697674] LNet: Removed LNI 10.1.2.31@tcp [ 825.523799] Key type .llcrypt unregistered [ 825.525466] Key type ._llcrypt unregistered [ 838.030313] Key type ._llcrypt registered [ 838.034444] Key type .llcrypt registered [ 839.546809] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:09:50 (1778839790) [ 839.886058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.904419] alg: No test for adler32 (adler32-zlib) [ 840.860897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.946108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 840.990335] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 841.922218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.940408] LNet: 38711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.950225] LNetError: 38711:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.968052] LNet: Removed LNI 192.168.201.55@tcp [ 841.984439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.057709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.55@tcp [ 842.100715] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 842.812168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.55@tcp [ 842.840239] LNet: Removed LNI 192.168.201.55@tcp [ 842.914632] LNet: 38846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.696541] Key type .llcrypt unregistered [ 843.700611] Key type ._llcrypt unregistered [ 855.690303] Key type ._llcrypt registered [ 855.692442] Key type .llcrypt registered [ 856.756353] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:10:07 (1778839807) [ 858.660451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg155-server [ 860.159700] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:10:11 (1778839811) [ 860.550524] Key type .llcrypt unregistered [ 860.552375] Key type ._llcrypt unregistered [ 865.304370] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 865.512749] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 865.521450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 865.523313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 868.001355] Key type ._llcrypt registered [ 868.003545] Key type .llcrypt registered [ 868.177937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.192467] alg: No test for adler32 (adler32-zlib) [ 869.197933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.250502] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 869.861429] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 871.528857] LNet: 40682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.535226] LNetError: 40682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.540375] LNet: Removed LNI 192.168.201.55@tcp1 [ 871.549597] LNet: Removed LNI 10.1.2.3@tcp2 [ 871.971517] Key type .llcrypt unregistered [ 871.973619] Key type ._llcrypt unregistered [ 881.496757] Key type ._llcrypt registered [ 881.499205] Key type .llcrypt registered [ 882.710194] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:10:33 (1778839833) [ 883.279276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 883.294862] alg: No test for adler32 (adler32-zlib) [ 884.254986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.356759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.398822] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 885.306598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.337765] LNet: 41650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.345874] LNetError: 41650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.373380] LNet: Removed LNI 192.168.201.55@tcp [ 885.394970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.456530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 885.482761] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 892.050837] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:10:43 (1778839843) [ 892.177348] LNet: 42234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.181704] LNetError: 42234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.211760] LNet: Removed LNI 192.168.201.55@tcp [ 892.836294] Key type .llcrypt unregistered [ 892.838188] Key type ._llcrypt unregistered [ 898.801042] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 898.902105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 898.908317] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 898.918549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 900.525439] Key type ._llcrypt registered [ 900.527266] Key type .llcrypt registered [ 900.741123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 900.770593] alg: No test for adler32 (adler32-zlib) [ 901.709114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 901.777197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 901.805770] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 902.604450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 902.618662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 904.853932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.882158] LNet: 42977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.888641] LNetError: 42977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.901769] LNet: Removed LNI 192.168.201.55@tcp [ 904.934170] LNet: Removed LNI 10.1.2.3@tcp [ 904.971721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.037460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 905.055777] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 905.060087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 905.571533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.611605] LNet: 43004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.623771] LNetError: 43004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.626746] LNetError: 43004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 905.641838] LNet: Removed LNI 192.168.201.55@tcp [ 905.664910] LNet: Removed LNI 10.1.2.3@tcp [ 905.722686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.791120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 905.817824] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 905.824386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.130304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 906.147066] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 906.967774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.979621] LNet: 43166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.989436] LNetError: 43166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.003641] LNetError: 43166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 907.013520] LNet: Removed LNI 192.168.201.55@tcp [ 907.015237] LNet: Removed LNI 10.1.2.3@tcp [ 907.036233] LNet: Removed LNI 192.168.201.55@tcp2 [ 907.075815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.164237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 907.205716] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 907.214688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.235864] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 914.211859] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:11:05 (1778839865) [ 914.328454] LNet: 43757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.340637] LNetError: 43757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.345431] LNetError: 43757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 914.352871] LNet: Removed LNI 192.168.201.55@tcp [ 914.355302] LNet: Removed LNI 10.1.2.3@tcp [ 914.363950] LNet: Removed LNI 192.168.201.55@tcp2 [ 914.957887] Key type .llcrypt unregistered [ 914.959748] Key type ._llcrypt unregistered [ 921.390377] Key type ._llcrypt registered [ 921.392285] Key type .llcrypt registered [ 921.608780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.621975] alg: No test for adler32 (adler32-zlib) [ 922.626869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.684511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.716123] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 923.569479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 923.615300] LNet: 44157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.635135] LNetError: 44157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.652170] LNet: Removed LNI 192.168.201.55@tcp [ 923.706945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.766352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 923.798572] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 924.278747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.294793] LNet: 44187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.302247] LNetError: 44187:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.317957] LNet: Removed LNI 192.168.201.55@tcp [ 924.345291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.397612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 924.418893] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 925.062528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.144249] LNet: Removed LNI 192.168.201.55@tcp [ 925.173187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.243684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 925.275180] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 925.776938] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.023301] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.054449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.064218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.530879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.568447] LNet: 44296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.573735] LNet: 44296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 928.586386] LNetError: 44296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.590367] LNetError: 44296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 928.613992] LNet: Removed LNI 192.168.201.55@tcp [ 928.648746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.740651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 928.776326] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 934.985013] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:11:26 (1778839886) [ 935.209699] LNet: 44890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.217775] LNetError: 44890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.243506] LNet: Removed LNI 192.168.201.55@tcp [ 935.956985] Key type .llcrypt unregistered [ 935.958629] Key type ._llcrypt unregistered [ 941.669742] Key type ._llcrypt registered [ 941.673903] Key type .llcrypt registered [ 941.862546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.883323] alg: No test for adler32 (adler32-zlib) [ 942.841062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.920426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 942.948768] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 943.731794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.756847] LNet: 45291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.764266] LNetError: 45291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.784154] LNet: Removed LNI 192.168.201.55@tcp [ 943.826346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.915233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 943.948927] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 950.073255] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:11:41 (1778839901) [ 950.419686] LNet: 45883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.423881] LNetError: 45883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.441133] LNet: Removed LNI 192.168.201.55@tcp [ 950.999665] Key type .llcrypt unregistered [ 951.001316] Key type ._llcrypt unregistered [ 955.963975] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.179199] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.186391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.195490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 958.667604] Key type ._llcrypt registered [ 958.669590] Key type .llcrypt registered [ 958.900241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.938459] alg: No test for adler32 (adler32-zlib) [ 959.942651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.022989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.061861] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 960.947762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.964765] LNet: 46333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.970350] LNetError: 46333:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.995496] LNet: Removed LNI 192.168.201.55@tcp [ 961.010686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.067090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 961.097505] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 967.594835] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:11:58 (1778839918) [ 967.728918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.765201] LNet: 46926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.776528] LNetError: 46926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.787721] LNet: Removed LNI 192.168.201.55@tcp [ 967.813379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.962222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 973.814773] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:12:04 (1778839924) [ 973.902823] LNet: 47514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.621061] Key type .llcrypt unregistered [ 974.627044] Key type ._llcrypt unregistered [ 980.274701] Key type ._llcrypt registered [ 980.276515] Key type .llcrypt registered [ 980.528843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.569032] alg: No test for adler32 (adler32-zlib) [ 981.486300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.577969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.612032] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 982.215593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.236507] LNet: 47920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.241613] LNetError: 47920:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.260514] LNet: Removed LNI 192.168.201.55@tcp [ 982.284555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.339677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 982.362830] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 988.324915] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:12:19 (1778839939) [ 988.482469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.526282] LNet: 48510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.542690] LNetError: 48510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.577181] LNet: Removed LNI 192.168.201.55@tcp [ 988.621864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.705601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 994.450390] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:12:25 (1778839945) [ 996.040499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 997.603452] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:12:28 (1778839948) [ 998.962793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1000.299995] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:12:31 (1778839951) [ 1001.557651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1002.698462] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:12:34 (1778839954) [ 1004.170338] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1005.604576] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:12:36 (1778839956) [ 1006.937291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1008.531536] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:12:39 (1778839959) [ 1008.873886] LNet: 50564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.636889] Key type .llcrypt unregistered [ 1009.641131] Key type ._llcrypt unregistered [ 1015.874792] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.949605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1015.957994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1015.962869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1018.511458] Key type ._llcrypt registered [ 1018.518837] Key type .llcrypt registered [ 1018.739805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1018.772661] alg: No test for adler32 (adler32-zlib) [ 1019.699496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.787777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.811701] LNet: Added LNI 192.168.201.55@tcp [8/512/0/180] [ 1019.816573] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1019.818063] LNet: 50885:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.915978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.936792] LNet: 50904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.940961] LNetError: 50904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.948777] LNet: Removed LNI 192.168.201.55@tcp [ 1019.951339] LNetError: 50904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.967495] LNet: Removed LNI 10.1.2.3@tcp [ 1019.984792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.035635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.056545] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1020.106921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.120617] LNet: 50928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.124952] LNetError: 50928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.134787] LNet: Removed LNI 192.168.201.55@tcp [ 1020.150029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.204717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.219839] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1020.224283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.226112] LNet: 50936:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.308195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.334168] LNet: 50954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.345508] LNetError: 50954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.349699] LNet: Removed LNI 192.168.201.55@tcp [ 1020.351742] LNetError: 50954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.366148] LNet: Removed LNI 10.1.2.3@tcp [ 1020.382838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.439310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.458825] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1020.467003] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.472295] LNet: 50962:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.685677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.704194] LNet: 50988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.712296] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.720895] LNet: Removed LNI 192.168.201.55@tcp [ 1020.723440] LNetError: 50988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.743849] LNet: Removed LNI 10.1.2.3@tcp [ 1020.769628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.823995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.838739] LNet: Added LNI 192.168.201.55@tcp [8/256/0/182] [ 1020.846669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1020.930708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.944328] LNet: 51016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.952817] LNetError: 51016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.960521] LNet: Removed LNI 192.168.201.55@tcp [ 1020.962903] LNetError: 51016:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.975890] LNet: Removed LNI 10.1.2.3@tcp [ 1020.993458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.038373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1021.052820] LNet: Added LNI 192.168.201.55@tcp [8/256/0/182] [ 1021.056475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1021.059474] LNet: 51024:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1021.115919] LNet: Removed LNI 192.168.201.55@tcp [ 1021.125957] LNet: Removed LNI 10.1.2.3@tcp [ 1021.619911] Key type .llcrypt unregistered [ 1021.621867] Key type ._llcrypt unregistered [ 1031.400380] Key type ._llcrypt registered [ 1031.405512] Key type .llcrypt registered [ 1032.666733] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:13:03 (1778839983) [ 1033.010380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.022525] alg: No test for adler32 (adler32-zlib) [ 1034.021399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.097155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.145024] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1035.020451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.55 [ 1035.045124] LNet: 51998:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1035.066402] LNet: Removed LNI 192.168.201.55@tcp [ 1041.258699] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:13:12 (1778839992) [ 1041.488499] LNet: 52565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.492534] LNetError: 52565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.524857] LNet: Removed LNI 192.168.201.55@tcp [ 1042.004618] Key type .llcrypt unregistered [ 1042.007298] Key type ._llcrypt unregistered [ 1047.576149] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.774177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1050.563552] Key type ._llcrypt registered [ 1050.565719] Key type .llcrypt registered [ 1050.738196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.747936] alg: No test for adler32 (adler32-zlib) [ 1052.123455] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.194091] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:13:29 (1778840009) [ 1058.400187] LNet: 53481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.407860] LNetError: 53481:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.428181] LNet: Removed LNI 10.1.2.3@tcp [ 1058.961600] Key type .llcrypt unregistered [ 1058.963264] Key type ._llcrypt unregistered [ 1065.435694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1065.735358] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1065.741128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1065.759262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.388099] Key type ._llcrypt registered [ 1067.390451] Key type .llcrypt registered [ 1067.578888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.597356] alg: No test for adler32 (adler32-zlib) [ 1068.526948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1075.637811] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:13:46 (1778840026) [ 1077.160465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1079.413543] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:13:50 (1778840030) [ 1079.541281] LNet: 54652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.555704] LNetError: 54652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1079.584891] LNet: Removed LNI 10.1.2.3@tcp [ 1080.140100] Key type .llcrypt unregistered [ 1080.144763] Key type ._llcrypt unregistered [ 1086.164662] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1086.391753] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1086.396749] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1086.401911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1089.336644] Key type ._llcrypt registered [ 1089.338713] Key type .llcrypt registered [ 1089.579372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.599345] alg: No test for adler32 (adler32-zlib) [ 1090.531947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.629843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1099.543488] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:14:10 (1778840050) [ 1100.106384] LNet: 55889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.112412] LNetError: 55889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1100.142013] LNet: Removed LNI 10.1.2.3@tcp [ 1101.132474] Key type .llcrypt unregistered [ 1101.138318] Key type ._llcrypt unregistered [ 1107.982871] Key type ._llcrypt registered [ 1107.986048] Key type .llcrypt registered [ 1108.276780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.306965] alg: No test for adler32 (adler32-zlib) [ 1109.278423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1109.298039] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1116.762750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1117.452929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_interrupt -r 1 [ 1117.481472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_interrupt -r 1 [ 1117.506664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1117.524153] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1117.765697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.789607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.825710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.849435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.867785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.886883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.904766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.927859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.941166] LNet: 56327:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.958294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.987192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.049582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.096791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.146401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.172674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.201440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.806917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_dropped -r 1 [ 1118.834636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_dropped -r 1 [ 1118.873749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1118.902349] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1119.103536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.130888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1119.158282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.185684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.210581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.240951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.269898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.298673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.313669] LNet: 56436:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.330840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.356423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.384965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.417394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.449912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.477078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.518844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.082847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_aborted -r 1 [ 1120.108361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_aborted -r 1 [ 1120.133783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1120.147394] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1120.325704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.350108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1120.370879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.402960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.437595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.467273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.512340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.541393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.544712] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1120.567265] LNet: 56545:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.595146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.625758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.666989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.700413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.734408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.771326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.825629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.465228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_no_route -r 1 [ 1121.496747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_no_route -r 1 [ 1121.522435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1121.770676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.794794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1121.818510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.845500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.869352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.893243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.912880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.962202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.990494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.022527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.065935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.106119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.154784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.213472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.247863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.799831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_timeout -r 1 [ 1122.853159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_timeout -r 1 [ 1122.908776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1122.927277] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1122.933538] LNet: Skipped 2 previous similar messages [ 1123.176910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.199564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1123.219279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.287025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.327883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.365314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.404542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.449881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.469770] LNet: 56763:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.475180] LNet: 56763:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1123.496786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.524738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.556945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.605695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.632879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.665605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.702630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.322075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_error -r 1 [ 1124.363668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_error -r 1 [ 1124.421409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1124.715298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.770843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1124.808807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.852095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.881842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.932110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.960200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.995162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.064332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.098731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.128829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.176432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.223980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.252887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.280765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.551216] LNet: 56914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.560902] LNetError: 56914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1126.631855] LNet: Removed LNI 192.168.201.55@tcp [ 1127.082712] Key type .llcrypt unregistered [ 1127.084477] Key type ._llcrypt unregistered [ 1141.410730] Key type ._llcrypt registered [ 1141.412348] Key type .llcrypt registered [ 1142.555941] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:14:53 (1778840093) [ 1143.464659] Key type .llcrypt unregistered [ 1143.466515] Key type ._llcrypt unregistered [ 1149.646448] Key type ._llcrypt registered [ 1149.648201] Key type .llcrypt registered [ 1149.971553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.015297] alg: No test for adler32 (adler32-zlib) [ 1150.917700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.935518] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1158.477495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1168.227338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.290227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1168.323180] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1169.859935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_interrupt -r 1 [ 1169.925401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_interrupt -r 1 [ 1169.981893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_interrupt -r 1 [ 1170.048259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_interrupt -r 1 [ 1170.106751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_interrupt -r 1 [ 1170.174827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_interrupt -r 1 [ 1170.215250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_interrupt -r 1 [ 1170.253290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_interrupt -r 1 [ 1170.297174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1170.316443] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1170.320984] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1170.583140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.644370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.709825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.755846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.799662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.854233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.890949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.932495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.946138] LNet: 58379:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.966105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.998734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.029855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.070779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.143288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.221873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.282190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.956518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_dropped -r 1 [ 1172.011469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_dropped -r 1 [ 1172.075594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_dropped -r 1 [ 1172.150079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_dropped -r 1 [ 1172.232131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_dropped -r 1 [ 1172.319898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_dropped -r 1 [ 1172.382725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_dropped -r 1 [ 1172.490617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_dropped -r 1 [ 1172.555480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1172.576676] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1172.583105] LNet: Skipped 2 previous similar messages [ 1172.853722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.907499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.953882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.003211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.044157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.112591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.168658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.198261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.214862] LNet: 58500:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.233759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.276687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.321991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.354389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.390270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.447443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.497477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.293887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_aborted -r 1 [ 1174.390829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_aborted -r 1 [ 1174.457790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_aborted -r 1 [ 1174.545947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_aborted -r 1 [ 1174.629291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_aborted -r 1 [ 1174.670206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_aborted -r 1 [ 1174.706448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_aborted -r 1 [ 1174.741876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_aborted -r 1 [ 1174.774657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1174.796876] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1174.802562] LNet: Skipped 2 previous similar messages [ 1175.045161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.075899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1175.119070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.149663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.195237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.241218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.268745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.294712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.310899] LNet: 58621:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.331934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.353057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.394102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.422565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.450994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.477882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.503813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.958633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_no_route -r 1 [ 1175.993984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_no_route -r 1 [ 1176.019265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_no_route -r 1 [ 1176.050284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_no_route -r 1 [ 1176.076808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_no_route -r 1 [ 1176.108300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_no_route -r 1 [ 1176.140901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_no_route -r 1 [ 1176.164968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_no_route -r 1 [ 1176.197504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1176.396934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.419411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.453245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.482435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.515575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.543337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.583883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.618284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.648686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.689881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.721682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.752101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.792745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.831648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.878216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.394115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_timeout -r 1 [ 1177.436388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_timeout -r 1 [ 1177.464956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_timeout -r 1 [ 1177.490193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_timeout -r 1 [ 1177.520358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_timeout -r 1 [ 1177.550736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_timeout -r 1 [ 1177.578812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_timeout -r 1 [ 1177.613124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_timeout -r 1 [ 1177.637687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1177.651776] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1177.657612] LNet: Skipped 5 previous similar messages [ 1177.832055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.870532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1177.908241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.939778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.974078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.006245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.029412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.051307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.062735] LNet: 58864:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.066887] LNet: 58864:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1178.080925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.109523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.134304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.161511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.202768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.243580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.282941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1178.756534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_error -r 1 [ 1178.789924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_error -r 1 [ 1178.838482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_error -r 1 [ 1178.871173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_error -r 1 [ 1178.899808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_error -r 1 [ 1178.927400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_error -r 1 [ 1178.961739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_error -r 1 [ 1178.989203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_error -r 1 [ 1179.015592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1179.248741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.269536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1179.295608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.319765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.353224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.383471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.418510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.453185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.475193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1179.501356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1179.533377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1179.562184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.591166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.616615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.656765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1184.382566] LNet: 59061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.390630] LNetError: 59061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.395339] LNet: Removed LNI 192.168.201.55@tcp [ 1184.415975] LNet: Removed LNI 192.168.201.55@tcp1 [ 1184.723757] Key type .llcrypt unregistered [ 1184.725895] Key type ._llcrypt unregistered [ 1197.002361] Key type ._llcrypt registered [ 1197.004895] Key type .llcrypt registered [ 1197.900749] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:15:49 (1778840149) [ 1198.628041] Key type .llcrypt unregistered [ 1198.629931] Key type ._llcrypt unregistered [ 1203.896568] Key type ._llcrypt registered [ 1203.903090] Key type .llcrypt registered [ 1204.064491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.073286] alg: No test for adler32 (adler32-zlib) [ 1204.983558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1205.004845] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1211.498245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1212.137287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_dropped -r 1 [ 1212.167447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_dropped -r 1 [ 1212.197296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1212.217550] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1212.421805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.464882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.505424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.540329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.572400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.620440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.646709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.687761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.707424] LNet: 60309:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.733990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.761743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.778493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.806518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.854656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.891193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.927641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.404569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_error -r 1 [ 1213.432526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_error -r 1 [ 1213.458675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1213.470541] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1213.644988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.678262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.700150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.729129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.752702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.774919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.798114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.822701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.838249] LNet: 60418:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.852179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.874754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.895531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.935386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.965675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.997107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.047413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1214.496074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_timeout -r 1 [ 1214.543605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_timeout -r 1 [ 1214.589274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1214.616331] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1214.909523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.944291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.988386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.020713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.052200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.085697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.112617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.141195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.155491] LNet: 60527:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.169655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.206875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.234348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.267926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.300477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.326483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.350761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.614211] LNet: 60568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.624910] LNetError: 60568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.656906] LNet: Removed LNI 192.168.201.55@tcp [ 1216.072267] Key type .llcrypt unregistered [ 1216.079233] Key type ._llcrypt unregistered [ 1232.229246] Key type ._llcrypt registered [ 1232.230727] Key type .llcrypt registered [ 1233.332274] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:16:24 (1778840184) [ 1233.908545] Key type .llcrypt unregistered [ 1233.910821] Key type ._llcrypt unregistered [ 1239.200721] Key type ._llcrypt registered [ 1239.202302] Key type .llcrypt registered [ 1239.455904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.473165] alg: No test for adler32 (adler32-zlib) [ 1240.380933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.416612] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1246.229974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1253.610870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.666230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.693590] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1255.009795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_dropped -r 1 [ 1255.041398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e remote_dropped -r 1 [ 1255.065438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_dropped -r 1 [ 1255.093725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e remote_dropped -r 1 [ 1255.125789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e remote_dropped -r 1 [ 1255.152479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e remote_dropped -r 1 [ 1255.177056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e remote_dropped -r 1 [ 1255.202474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e remote_dropped -r 1 [ 1255.228727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1255.242285] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1255.249826] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.155@tcp [ 1255.490150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.528465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1255.558395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.597660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.638890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.678699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.710105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.743251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.779659] LNet: 62034:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.820559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.850793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.900853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.947463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.007600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.055435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.118734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1256.157676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.650272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_error -r 1 [ 1256.673252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e remote_error -r 1 [ 1256.698923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_error -r 1 [ 1256.747138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e remote_error -r 1 [ 1256.802521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e remote_error -r 1 [ 1256.834486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e remote_error -r 1 [ 1256.871497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e remote_error -r 1 [ 1256.913764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e remote_error -r 1 [ 1256.963036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1256.984697] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1256.990485] LNet: Skipped 2 previous similar messages [ 1257.222698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.262097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1257.297522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.330230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.385698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.429033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.452638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.474233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.490215] LNet: 62157:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.515925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.554932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.583765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.606260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.629807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.679836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.749341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1257.833242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.403451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_timeout -r 1 [ 1258.448881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e remote_timeout -r 1 [ 1258.481365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_timeout -r 1 [ 1258.527336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e remote_timeout -r 1 [ 1258.576184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e remote_timeout -r 1 [ 1258.632495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e remote_timeout -r 1 [ 1258.696841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e remote_timeout -r 1 [ 1258.757429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e remote_timeout -r 1 [ 1258.814472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.155@tcp [ 1258.838875] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1258.844050] LNet: Skipped 1 previous similar message [ 1259.110767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.189526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1259.229098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.266987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.317637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.368847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.418526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.476407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.509621] LNet: 62281:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.532224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.578633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.626806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.723577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.774543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.803602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1259.836575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1259.861740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.049915] LNet: 62358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.056437] LNetError: 62358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.122140] LNet: Removed LNI 192.168.201.55@tcp [ 1265.123416] LNetError: 62358:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.145307] LNet: Removed LNI 192.168.201.55@tcp1 [ 1265.702868] Key type .llcrypt unregistered [ 1265.704407] Key type ._llcrypt unregistered [ 1278.996911] Key type ._llcrypt registered [ 1278.999205] Key type .llcrypt registered [ 1280.126937] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:17:11 (1778840231) [ 1281.152898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1282.265624] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:17:13 (1778840233) [ 1282.898204] Key type .llcrypt unregistered [ 1282.900200] Key type ._llcrypt unregistered [ 1288.361798] Key type ._llcrypt registered [ 1288.365924] Key type .llcrypt registered [ 1288.630285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.681645] alg: No test for adler32 (adler32-zlib) [ 1289.616245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.647049] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1295.966537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1296.673740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e network_timeout -r 1 [ 1296.709720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e network_timeout -r 1 [ 1296.743148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1296.763085] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1296.867837] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1297.028329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.056863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1297.088201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.122866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.151930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.178870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.211315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.236722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.252819] LNet: 63900:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.272932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.302879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.330303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.358188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.380928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.405734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.629609] LNet: 63939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.634305] LNetError: 63939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.653173] LNet: Removed LNI 192.168.201.55@tcp [ 1298.023384] Key type .llcrypt unregistered [ 1298.026493] Key type ._llcrypt unregistered [ 1313.730386] Key type ._llcrypt registered [ 1313.731937] Key type .llcrypt registered [ 1313.936934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1313.955673] alg: No test for adler32 (adler32-zlib) [ 1314.842859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1314.869731] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1321.748886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1329.780208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1329.815520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.830571] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1331.344184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e network_timeout -r 1 [ 1331.414985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e network_timeout -r 1 [ 1331.475051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e network_timeout -r 1 [ 1331.529247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e network_timeout -r 1 [ 1331.591927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e network_timeout -r 1 [ 1331.635572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e network_timeout -r 1 [ 1331.695333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e network_timeout -r 1 [ 1331.760886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e network_timeout -r 1 [ 1331.826361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1331.855108] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1332.118723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1332.155397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1332.176351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1332.200790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1332.222326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1332.254452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1332.280620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1332.309141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1332.320596] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1332.322429] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.155@tcp [ 1332.331526] LNet: 64817:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1332.347636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.375368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.403949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.428169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.453862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.482480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.491966] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1336.499300] LNet: Skipped 2 previous similar messages [ 1336.767357] LNet: 64891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.771641] LNetError: 64891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.778489] LNet: Removed LNI 192.168.201.55@tcp [ 1336.786965] LNet: Removed LNI 192.168.201.55@tcp1 [ 1337.171550] Key type .llcrypt unregistered [ 1337.177708] Key type ._llcrypt unregistered [ 1350.830615] Key type ._llcrypt registered [ 1350.832307] Key type .llcrypt registered [ 1351.862074] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:18:23 (1778840303) [ 1352.202669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1352.216554] alg: No test for adler32 (adler32-zlib) [ 1353.114832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.167123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1353.184726] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1353.747674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1353.760586] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1354.370627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1354.422619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1354.456644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1354.481203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1354.510724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1354.532767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1354.562641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.55@tcp [ 1354.580977] LNetError: 65988:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.55@tcp to recovery [ 1355.169099] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1355.169144] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1357.216647] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1358.010446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.314071] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1361.505660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1361.538570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1361.608590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.628096] LNet: 66095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.633814] LNetError: 66095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.656462] LNet: Removed LNI 192.168.201.55@tcp [ 1362.658985] LNetError: 66095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.676686] LNet: Removed LNI 192.168.201.55@tcp1 [ 1362.695702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.735847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.751590] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1363.365990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.387849] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1364.025836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1364.059756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.138147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1364.198980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1364.257890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1364.301871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1364.351694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1364.401268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.55@tcp [ 1364.423650] LNetError: 66368:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.55@tcp to recovery [ 1364.769000] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1367.931698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1370.912435] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1370.915951] LNet: Skipped 1 previous similar message [ 1375.758510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1375.787822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1375.816333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.535796] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:18:53 (1778840333) [ 1382.899635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.939629] LNet: 67084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.945560] LNetError: 67084:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.969522] LNet: Removed LNI 192.168.201.55@tcp [ 1383.988465] LNet: Removed LNI 192.168.201.55@tcp1 [ 1384.041631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.139748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.187178] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1384.821420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1384.845424] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1385.664732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1385.725400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1385.751364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1385.777642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1385.809344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1385.834518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1385.864384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.55@tcp --health 0 [ 1385.948785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1386.144923] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.55@tcp [ 1386.146065] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1386.167023] LNet: Skipped 1 previous similar message [ 1401.271192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1401.397080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1401.422744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.451498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.55@tcp --health 500 [ 1401.559276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1408.220857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1408.275742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1408.300404] LNet: 67577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1408.306462] LNetError: 67577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.313294] LNetError: 67577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1408.317608] LNet: Removed LNI 192.168.201.55@tcp [ 1408.324147] LNet: Removed LNI 192.168.201.55@tcp1 [ 1408.351362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.392566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1408.407728] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1409.072190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1409.091362] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1409.822177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1409.864646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.920570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1409.950465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1409.985777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1410.019538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1410.051703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1410.085692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.55@tcp --health 0 [ 1411.426688] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1411.429133] LNet: Skipped 3 previous similar messages [ 1412.493015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1422.392853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1422.411763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1422.431912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1427.109783] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 06:19:38 (1778840378) [ 1427.398975] LNet: 68634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.401731] LNetError: 68634:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.404576] LNetError: 68634:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1427.408571] LNet: Removed LNI 192.168.201.55@tcp [ 1427.414725] LNet: Removed LNI 192.168.201.55@tcp1 [ 1427.851655] Key type .llcrypt unregistered [ 1427.852955] Key type ._llcrypt unregistered [ 1432.699189] Key type ._llcrypt registered [ 1432.701215] Key type .llcrypt registered [ 1432.867216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1432.880950] alg: No test for adler32 (adler32-zlib) [ 1433.799734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1433.822604] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1440.733992] Lustre: DEBUG MARKER: Initial discovery [ 1440.776225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.155@tcp [ 1442.837449] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1442.887445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1442.930544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.155@tcp [ 1442.956644] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1442.975287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1443.017700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.155@tcp -r 1 -m GET -l 3 [ 1443.041115] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1443.091101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.155@tcp -r 1 -m GET -e local_error [ 1443.118980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.155@tcp -r 1 -m PUT -l 6 [ 1444.157870] Lustre: DEBUG MARKER: Force oleg155-client.virtnet to discover 192.168.201.155@tcp (in background) [ 1444.387382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.155@tcp [ 1445.614739] Lustre: DEBUG MARKER: Force oleg155-server to discover 192.168.201.55@tcp [ 1447.457243] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1448.134233] Lustre: DEBUG MARKER: Wait for 69397 [ 1449.629539] Lustre: DEBUG MARKER: Finished wait on 69397 [ 1449.687816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1449.719977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1449.720278] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1449.959569] LNet: 69845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.969970] LNetError: 69845:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.993415] LNet: Removed LNI 192.168.201.55@tcp [ 1450.321667] Key type .llcrypt unregistered [ 1450.323830] Key type ._llcrypt unregistered [ 1462.145884] Key type ._llcrypt registered [ 1462.148091] Key type .llcrypt registered [ 1463.066682] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 06:20:14 (1778840414) [ 1463.455577] Key type .llcrypt unregistered [ 1463.457781] Key type ._llcrypt unregistered [ 1467.876627] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1468.045584] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1468.049715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1468.052432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1469.383863] Key type ._llcrypt registered [ 1469.386189] Key type .llcrypt registered [ 1469.474421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1469.484312] alg: No test for adler32 (adler32-zlib) [ 1470.400602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.439047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1470.458645] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1470.990205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1471.004763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1478.048485] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 06:20:29 (1778840429) [ 1478.309439] LNet: 71912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.318623] LNetError: 71912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.328451] LNet: Removed LNI 192.168.201.55@tcp [ 1478.358938] LNet: Removed LNI 10.1.2.3@tcp [ 1478.970459] Key type .llcrypt unregistered [ 1478.972713] Key type ._llcrypt unregistered [ 1484.307467] Key type ._llcrypt registered [ 1484.309050] Key type .llcrypt registered [ 1484.556899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1484.572474] alg: No test for adler32 (adler32-zlib) [ 1485.426660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.470552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.486455] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1486.067785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1486.099580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1486.945147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.55@tcp [ 1488.033526] LNetError: Unexpected error -22 connecting to 192.168.201.55@tcp at host 192.168.201.55:988 [ 1492.332072] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 06:20:43 (1778840443) [ 1492.421442] LNet: 73035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.436713] LNetError: 73035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.440442] LNet: Removed LNI 192.168.201.55@tcp [ 1492.446945] LNet: Removed LNI 10.1.2.3@tcp [ 1492.885534] Key type .llcrypt unregistered [ 1492.888255] Key type ._llcrypt unregistered [ 1498.349595] Key type ._llcrypt registered [ 1498.352994] Key type .llcrypt registered [ 1498.504609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1498.521459] alg: No test for adler32 (adler32-zlib) [ 1499.402763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.474537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.507149] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1500.146204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1500.169683] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 1500.831585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.55@tcp1 --nid 192.168.201.55@tcp2 [ 1504.264980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp1 192.168.201.55@tcp2 [ 1504.282815] LNetError: 73877:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.55@tcp1 to 192.168.201.55@tcp2 - Route table may be misconfigured [ 1504.303699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp2 192.168.201.55@tcp1 [ 1510.263795] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 06:21:01 (1778840461) [ 1510.509673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.538399] LNet: 74450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.547001] LNetError: 74450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.584450] LNet: Removed LNI 192.168.201.55@tcp1 [ 1511.589260] LNetError: 74450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.602089] LNet: Removed LNI 192.168.201.55@tcp2 [ 1511.636945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.716968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.738699] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1512.253285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.268069] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1512.934411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1512.959476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.55@tcp [ 1512.989228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp1 192.168.201.55@tcp1 [ 1513.019294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.55@tcp -d 192.168.201.55@tcp -e network_timeout [ 1513.046976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -e network_timeout [ 1513.075238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -e network_timeout [ 1513.106308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -e network_timeout [ 1513.158214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1513.189354] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1513.215515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1513.290389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1513.355613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1513.696165] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1518.318979] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 06:21:09 (1778840469) [ 1518.425193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.453430] LNet: 75306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.464382] LNetError: 75306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.469592] LNet: Removed LNI 192.168.201.55@tcp [ 1519.530305] LNet: Removed LNI 192.168.201.55@tcp1 [ 1519.550681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.619757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1519.844234] LNet: 75350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.144968] Key type .llcrypt unregistered [ 1520.149707] Key type ._llcrypt unregistered [ 1528.773472] Key type ._llcrypt registered [ 1528.775622] Key type .llcrypt registered [ 1529.975834] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 06:21:21 (1778840481) [ 1530.193411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.199402] alg: No test for adler32 (adler32-zlib) [ 1531.057200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.078860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1533.367746] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 06:21:24 (1778840484) [ 1533.556670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.576854] LNet: 76447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.597760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.637588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.659776] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1534.175823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1534.190859] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1534.868600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1534.893042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp1 [ 1534.922748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp1 [ 1539.809483] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 06:21:30 (1778840490) [ 1541.195930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg155-server [ 1542.612788] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 06:21:33 (1778840493) [ 1544.099576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg155-server [ 1545.436968] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 06:21:36 (1778840496) [ 1546.969300] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg155-server [ 1548.448536] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 06:21:39 (1778840499) [ 1550.038453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg155-server [ 1551.215759] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 06:21:42 (1778840502) [ 1552.432473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg155-server [ 1554.087149] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 06:21:45 (1778840505) [ 1555.797711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg155-server [ 1557.151471] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 06:21:48 (1778840508) [ 1558.611478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg155-server [ 1559.893243] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 06:21:51 (1778840511) [ 1561.299883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg155-server [ 1562.499239] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 06:21:53 (1778840513) [ 1563.877176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg155-server [ 1565.093955] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 06:21:56 (1778840516) [ 1565.197885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.249340] LNet: 80319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.258978] LNetError: 80319:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.266405] LNet: Removed LNI 192.168.201.55@tcp [ 1566.312208] LNet: Removed LNI 192.168.201.55@tcp1 [ 1566.347911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.402248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.429658] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1567.021211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1567.103914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1567.244510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.265443] LNet: 80471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.269963] LNetError: 80471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.274817] LNetError: 80471:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1567.291817] LNet: Removed LNI 192.168.201.55@tcp [ 1567.319582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.375260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1567.396775] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1568.087568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1568.246944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.320411] LNet: Removed LNI 192.168.201.55@tcp [ 1569.355429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.429914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.449474] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1570.064387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1570.130969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1570.328237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.362226] LNet: 80763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.374580] LNet: 80763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1570.381094] LNetError: 80763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.388974] LNetError: 80763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1570.429312] LNet: Removed LNI 192.168.201.55@tcp [ 1570.462846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.556410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1570.589505] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1571.420202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1571.665483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.711535] LNet: Removed LNI 192.168.201.55@tcp [ 1572.753922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.823234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.856674] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1573.540508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1573.634701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1573.839958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.861962] LNetError: 81056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.875518] LNetError: 81056:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1573.903424] LNet: Removed LNI 192.168.201.55@tcp [ 1573.942767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.992465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1574.016385] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1574.749939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1574.952631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.991709] LNet: 81201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.001691] LNet: 81201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1576.041982] LNet: Removed LNI 192.168.201.55@tcp [ 1576.094979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.158587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.183057] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1576.814888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1576.900990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1577.127754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.215669] LNet: Removed LNI 192.168.201.55@tcp [ 1578.251819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.327030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1578.363745] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1579.085926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1579.287766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.315713] LNetError: 81494:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1579.320904] LNetError: 81494:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1580.400579] LNet: Removed LNI 192.168.201.55@tcp [ 1580.430412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.476621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.503723] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1581.160759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1581.252676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1581.476735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.566568] LNet: Removed LNI 192.168.201.55@tcp [ 1582.587492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.638259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1582.652894] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1583.219587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1583.403988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.435694] LNet: 81786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.443477] LNet: 81786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1584.486504] LNet: Removed LNI 192.168.201.55@tcp [ 1584.515564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.601664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.628994] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1585.289601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1585.358633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1585.575917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.671495] LNet: Removed LNI 192.168.201.55@tcp [ 1586.708403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.790460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1586.812399] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1587.632356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1587.979386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.002458] LNetError: 82079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.010548] LNetError: 82079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1589.048675] LNet: Removed LNI 192.168.201.55@tcp [ 1589.088251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.171246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.192397] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1589.753706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1589.798049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.823506] LNet: Removed LNI 192.168.201.55@tcp [ 1589.847180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.920573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.945924] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1590.549669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1596.328595] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 06:22:27 (1778840547) [ 1596.476141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.510286] LNet: Removed LNI 192.168.201.55@tcp [ 1596.530366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.581790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1596.593087] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/180] [ 1597.600894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.642870] LNet: Removed LNI 192.168.201.55@tcp231 [ 1597.656978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.728987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1597.764355] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/180] [ 1597.771505] LNet: 83061:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1598.250626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1598.307378] LNet: Removed LNI 192.168.201.55@tcp231 [ 1598.342815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1598.378770] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/180] [ 1599.322199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.352115] LNet: Removed LNI 192.168.201.55@tcp231 [ 1599.368069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.413181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1599.430554] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/0] [ 1599.434200] LNet: 83214:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1599.785918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1599.818049] LNet: Removed LNI 192.168.201.55@tcp231 [ 1599.831270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1599.850618] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/0] [ 1600.663211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.678835] LNet: 83357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1600.685258] LNet: 83357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1600.711830] LNet: Removed LNI 192.168.201.55@tcp231 [ 1600.733738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.807463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1600.825603] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/60] [ 1600.831461] LNet: 83365:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1601.128545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1601.141195] LNet: Removed LNI 192.168.201.55@tcp231 [ 1601.151767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1601.174335] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/60] [ 1601.963508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.998172] LNet: Removed LNI 192.168.201.55@tcp231 [ 1602.031861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.093847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1602.115302] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/180] [ 1602.412361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1602.438826] LNet: Removed LNI 192.168.201.55@tcp231 [ 1602.449867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1602.474843] LNet: Added LNI 192.168.201.55@tcp231 [8/256/0/180] [ 1608.535389] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 06:22:39 (1778840559) [ 1608.697055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.722312] LNetError: 84227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.726865] LNetError: 84227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1608.737220] LNet: Removed LNI 192.168.201.55@tcp231 [ 1608.748703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.786270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1608.811514] LNet: Added LNI 192.168.201.55@tcp232 [8/256/0/180] [ 1609.383780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.420699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.452048] LNet: Removed LNI 192.168.201.55@tcp232 [ 1609.479382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1609.513291] LNet: Added LNI 192.168.201.55@tcp232 [8/256/0/180] [ 1610.111541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1610.151394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1610.187550] LNet: Removed LNI 192.168.201.55@tcp232 [ 1610.201962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1610.235050] LNet: Added LNI 192.168.201.55@tcp232 [8/256/0/180] [ 1610.890875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1610.920149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1610.952944] LNet: Removed LNI 192.168.201.55@tcp232 [ 1610.981591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1611.019403] LNet: Added LNI 192.168.201.55@tcp232 [8/256/0/180] [ 1611.669248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1611.705356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.730190] LNet: Removed LNI 192.168.201.55@tcp232 [ 1611.740742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1616.451237] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 06:22:47 (1778840567) [ 1617.436651] Key type .llcrypt unregistered [ 1617.439119] Key type ._llcrypt unregistered [ 1622.065454] Key type ._llcrypt registered [ 1622.068837] Key type .llcrypt registered [ 1622.230909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1622.244443] alg: No test for adler32 (adler32-zlib) [ 1623.125107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1623.151540] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1628.677996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1634.183892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.224602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1634.241554] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1635.133215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_interrupt -r 2 [ 1635.157327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_interrupt -r 2 [ 1635.189832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_interrupt -r 2 [ 1635.215696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_interrupt -r 2 [ 1635.240250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_interrupt -r 2 [ 1635.273756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_interrupt -r 2 [ 1635.303779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_interrupt -r 2 [ 1635.330430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_interrupt -r 2 [ 1635.358934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1635.373638] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1635.376848] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1635.540913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1635.566607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1635.589404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1635.619561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1635.637476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1635.661688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1635.686589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1635.712338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1635.729460] LNet: 86016:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.749657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1635.775107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1635.794141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1635.814410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1635.838856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1635.858690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.268658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_dropped -r 2 [ 1636.300710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_dropped -r 2 [ 1636.328273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_dropped -r 2 [ 1636.357801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_dropped -r 2 [ 1636.376613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_dropped -r 2 [ 1636.394210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_dropped -r 2 [ 1636.422709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_dropped -r 2 [ 1636.442673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_dropped -r 2 [ 1636.471287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1636.486212] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1636.695206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.717564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1636.744992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.765210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.800552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.833354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.853709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1636.878307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1636.893952] LNet: 86136:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1636.908206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1636.933747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1636.960323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.981589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.995678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1637.021656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1637.315943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_aborted -r 2 [ 1637.342687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_aborted -r 2 [ 1637.372708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_aborted -r 2 [ 1637.399565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_aborted -r 2 [ 1637.429869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_aborted -r 2 [ 1637.467773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_aborted -r 2 [ 1637.499280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_aborted -r 2 [ 1637.525382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_aborted -r 2 [ 1637.561626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1637.583032] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1637.766942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1637.793292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1637.818339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1637.847924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1637.867139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1637.890829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1637.915102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1637.943463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1637.956581] LNet: 86256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1637.967292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1637.996219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.043750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.070390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1638.110815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1638.149624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1638.571454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_no_route -r 2 [ 1638.593497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_no_route -r 2 [ 1638.628984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_no_route -r 2 [ 1638.660439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_no_route -r 2 [ 1638.695768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_no_route -r 2 [ 1638.724551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_no_route -r 2 [ 1638.754940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_no_route -r 2 [ 1638.784953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_no_route -r 2 [ 1638.817536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1639.033613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1639.051376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1639.068858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1639.090924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1639.120598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1639.143325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1639.164592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1639.186342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1639.209788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1639.238536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1639.263634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1639.284181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1639.311084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.343363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.764701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e local_timeout -r 2 [ 1639.791811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e local_timeout -r 2 [ 1639.822163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e local_timeout -r 2 [ 1639.847943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e local_timeout -r 2 [ 1639.875837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e local_timeout -r 2 [ 1639.902734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e local_timeout -r 2 [ 1639.930504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e local_timeout -r 2 [ 1639.956894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e local_timeout -r 2 [ 1639.985509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1640.009412] LNet: There was an unexpected network error while writing to 192.168.201.155: rc = -22 [ 1640.013890] LNet: Skipped 1 previous similar message [ 1640.184447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.219125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1640.245112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.271745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.292513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.319398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.343786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.363913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.377602] LNet: 86497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1640.382160] LNet: 86497:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1640.394422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.416653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.446072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.467908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1640.491220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1640.520529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1640.970722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp -m GET -e remote_dropped -r 2 [ 1640.998772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.155@tcp1 -m GET -e remote_dropped -r 2 [ 1641.028638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp -m GET -e remote_dropped -r 2 [ 1641.058682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp -d 192.168.201.55@tcp1 -m GET -e remote_dropped -r 2 [ 1641.083655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp -m GET -e remote_dropped -r 2 [ 1641.108531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.155@tcp1 -m GET -e remote_dropped -r 2 [ 1641.135313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp -m GET -e remote_dropped -r 2 [ 1641.177854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.55@tcp1 -d 192.168.201.55@tcp1 -m GET -e remote_dropped -r 2 [ 1641.222488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.55@tcp 192.168.201.155@tcp [ 1641.237759] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.155@tcp [ 1641.386282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1641.407445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1641.431829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1641.460401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1641.490834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1641.506366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1641.522635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1641.541773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1641.570519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1641.594123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1641.620441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1641.648703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1641.675288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1641.714277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.641649] LNet: 86691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.648481] LNetError: 86691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.657906] LNet: Removed LNI 192.168.201.55@tcp [ 1645.664894] LNet: Removed LNI 192.168.201.55@tcp1 [ 1645.995538] Key type .llcrypt unregistered [ 1645.997740] Key type ._llcrypt unregistered [ 1659.510420] Key type ._llcrypt registered [ 1659.514863] Key type .llcrypt registered [ 1660.775854] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 06:23:31 (1778840611) [ 1661.148855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1661.165935] alg: No test for adler32 (adler32-zlib) [ 1662.113528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.187606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.225504] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1662.920326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1662.948288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.55@tcp [ 1662.961935] LNetError: 87669:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.55@tcp to recovery [ 1663.040544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.55@tcp [ 1663.125554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.050951] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 06:23:39 (1778840619) [ 1668.087806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.106989] LNet: 88251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.111929] LNetError: 88251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.123821] LNet: Removed LNI 192.168.201.55@tcp [ 1668.139642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.189681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.205102] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1668.610744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1668.626857] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1669.383494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.55@tcp [ 1669.394594] LNetError: 88525:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.55@tcp to recovery [ 1670.241543] LNet: 1 local NIs in recovery (showing 1): 192.168.201.55@tcp [ 1670.245682] LNet: There was an unexpected network error while writing to 192.168.201.55: rc = -22 [ 1670.583630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.55@tcp [ 1678.511182] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 06:23:49 (1778840629) [ 1680.136377] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg155-server [ 1682.079272] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 06:23:53 (1778840633) [ 1684.167390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg155-server [ 1685.551256] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 06:23:56 (1778840636) [ 1686.837462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg155-server [ 1688.180718] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 06:23:59 (1778840639) [ 1688.339802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.360189] LNet: 90155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.364776] LNetError: 90155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.376147] LNet: Removed LNI 192.168.201.55@tcp [ 1689.377767] LNetError: 90155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.390675] LNet: Removed LNI 192.168.201.55@tcp1 [ 1689.411836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.447648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.462228] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1689.935720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1689.970258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1690.018268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1690.088650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1690.124941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1690.161982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1695.775826] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 06:24:06 (1778840646) [ 1695.846815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.876515] LNet: 90874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.881539] LNetError: 90874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.898234] LNet: Removed LNI 192.168.201.55@tcp [ 1695.922598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.959463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.980991] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1702.205555] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 06:24:13 (1778840653) [ 1702.561941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.583735] LNet: 91583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.590476] LNetError: 91583:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.618683] LNet: Removed LNI 192.168.201.55@tcp [ 1702.657638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.728580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.765697] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1709.310389] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 06:24:20 (1778840660) [ 1710.460482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1711.745983] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 06:24:23 (1778840663) [ 1712.045520] LNet: 92618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.050514] LNetError: 92618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.063052] LNet: Removed LNI 192.168.201.55@tcp [ 1712.634562] Key type .llcrypt unregistered [ 1712.638278] Key type ._llcrypt unregistered [ 1717.299533] Key type ._llcrypt registered [ 1717.301799] Key type .llcrypt registered [ 1717.439183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1717.451400] alg: No test for adler32 (adler32-zlib) [ 1718.345990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1718.371459] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1724.611272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1728.344550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.155@tcp [ 1728.662890] LNet: 93035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.669694] LNetError: 93035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.684350] LNet: Removed LNI 192.168.201.55@tcp [ 1728.943384] Key type .llcrypt unregistered [ 1728.945070] Key type ._llcrypt unregistered [ 1738.307455] Key type ._llcrypt registered [ 1738.309378] Key type .llcrypt registered [ 1739.424744] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1741.196888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1742.416776] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 06:24:53 (1778840693) [ 1743.665818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1745.098309] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 06:24:56 (1778840696) [ 1746.454388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg155-server [ 1748.188162] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 06:24:59 (1778840699) [ 1749.563553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg155-server [ 1751.176177] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 06:25:02 (1778840702) [ 1752.463614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1752.490968] alg: No test for adler32 (adler32-zlib) [ 1753.465715] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 1753.588872] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1755.232842] Key type lgssc registered [ 1756.499233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1771.173952] Key type lgssc unregistered [ 1771.330531] LNet: 96210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.335193] LNetError: 96210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1771.348393] LNet: Removed LNI 192.168.201.55@tcp [ 1771.789402] Key type .llcrypt unregistered [ 1771.790907] Key type ._llcrypt unregistered [ 1781.612380] Key type ._llcrypt registered [ 1781.614349] Key type .llcrypt registered [ 1782.529895] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 06:25:33 (1778840733) [ 1782.748191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.756157] alg: No test for adler32 (adler32-zlib) [ 1783.609522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.656947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1783.687979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1783.711327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1783.742856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1788.086059] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 06:25:39 (1778840739) [ 1788.173440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.188615] LNet: 97625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.204061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.258029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.285096] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1788.894616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.908888] LNet: 97773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.912752] LNetError: 97773:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.921124] LNet: Removed LNI 192.168.201.55@tcp [ 1788.934524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.984841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1788.998596] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1789.304434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.337119] LNet: Removed LNI 192.168.201.55@tcp [ 1789.357652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.401723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1789.428440] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1793.798951] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 06:25:45 (1778840745) [ 1793.866932] LNet: 98382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.869789] LNet: 98382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1793.878477] LNetError: 98382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.882171] LNetError: 98382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1793.905989] LNet: Removed LNI 192.168.201.55@tcp [ 1794.419827] Key type .llcrypt unregistered [ 1794.421442] Key type ._llcrypt unregistered [ 1798.761652] Key type ._llcrypt registered [ 1798.764440] Key type .llcrypt registered [ 1798.942915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1798.952113] alg: No test for adler32 (adler32-zlib) [ 1799.865832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.923906] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1799.927188] LNet: *** cfs_fail_loc=e003, val=100*** [ 1800.485385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1800.504086] LNet: *** cfs_fail_loc=e003, val=100*** [ 1800.506558] LNet: Skipped 2 previous similar messages [ 1800.513416] Ping reply received with mapping: [ 1800.518817] found metadata reply NI[192.168.201.55@tcp] [ 1800.521237] 37 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.524121] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.526509] 5a 5a 5a 5a 5a 5a 5a 5a [ 1800.629619] LNet: 98789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.634741] LNetError: 98789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.650540] LNet: Removed LNI 192.168.201.55@tcp [ 1801.239616] Key type .llcrypt unregistered [ 1801.244349] Key type ._llcrypt unregistered [ 1809.258766] Key type ._llcrypt registered [ 1809.260396] Key type .llcrypt registered [ 1810.183350] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 06:26:01 (1778840761) [ 1810.520324] Key type .llcrypt unregistered [ 1810.521989] Key type ._llcrypt unregistered [ 1815.284418] Key type ._llcrypt registered [ 1815.286423] Key type .llcrypt registered [ 1815.485646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1815.502717] alg: No test for adler32 (adler32-zlib) [ 1816.624635] LNetError: 99833:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1816.959464] Key type .llcrypt unregistered [ 1816.962507] Key type ._llcrypt unregistered [ 1820.835527] Key type ._llcrypt registered [ 1820.837948] Key type .llcrypt registered [ 1821.799751] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 06:26:13 (1778840773) [ 1822.204586] Key type .llcrypt unregistered [ 1822.208204] Key type ._llcrypt unregistered [ 1826.198965] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1826.210369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1828.657720] Key type ._llcrypt registered [ 1828.662372] Key type .llcrypt registered [ 1828.819318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.832774] alg: No test for adler32 (adler32-zlib) [ 1829.741907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.793894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1837.790391] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 06:26:28 (1778840788) [ 1837.991599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.005685] LNet: 101613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.010790] LNetError: 101613:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1838.030414] LNet: Removed LNI 10.1.2.3@tcp [ 1838.055197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.700306] Key type .llcrypt unregistered [ 1838.701890] Key type ._llcrypt unregistered [ 1843.686652] Key type ._llcrypt registered [ 1843.690716] Key type .llcrypt registered [ 1843.829212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.839330] alg: No test for adler32 (adler32-zlib) [ 1844.762253] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1844.813214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1844.944492] LNet: 101902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.949678] LNetError: 101902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.958113] LNet: Removed LNI 192.168.201.55@tcp [ 1845.600501] Key type .llcrypt unregistered [ 1845.606635] Key type ._llcrypt unregistered [ 1853.602698] Key type ._llcrypt registered [ 1853.604099] Key type .llcrypt registered [ 1854.465833] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 06:26:45 (1778840805) [ 1855.044406] Key type .llcrypt unregistered [ 1855.046411] Key type ._llcrypt unregistered [ 1860.189196] Key type ._llcrypt registered [ 1860.191309] Key type .llcrypt registered [ 1860.322850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1860.333411] alg: No test for adler32 (adler32-zlib) [ 1865.561241] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 06:26:56 (1778840816) [ 1865.643439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.678527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.745180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.792170] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1866.287665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1866.312129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1866.342318] LNet: Removed LNI 192.168.201.55@tcp [ 1866.529493] LNet: 103652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.894409] Key type .llcrypt unregistered [ 1866.896559] Key type ._llcrypt unregistered [ 1875.182195] Key type ._llcrypt registered [ 1875.183888] Key type .llcrypt registered [ 1876.279892] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 06:27:07 (1778840827) [ 1876.710463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1876.720890] alg: No test for adler32 (adler32-zlib) [ 1877.624883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.677530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.697575] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1878.435974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1878.465666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.55@tcp [ 1878.485655] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1878.489179] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1878.491676] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1878, 0, 0 [ 1878.495370] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1878.496949] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1878.499332] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1878.502036] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1878, 0, 0 [ 1878.505631] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1878.508194] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1878.511554] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1878.514380] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1878, 0, 1928 [ 1878.519311] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1878.521448] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1878.523790] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1878.528491] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1878, 0, 1928 [ 1878.531563] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1883.411715] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 06:27:14 (1778840834) [ 1883.683903] LNet: 105191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1883.690518] LNetError: 105191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.709915] LNet: Removed LNI 192.168.201.55@tcp [ 1885.168251] Key type .llcrypt unregistered [ 1885.169400] Key type ._llcrypt unregistered [ 1889.724971] Key type ._llcrypt registered [ 1889.727570] Key type .llcrypt registered [ 1889.869622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1889.887375] alg: No test for adler32 (adler32-zlib) [ 1890.843960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1890.868651] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1897.261976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 1903.798253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.837394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1903.867283] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 1904.829798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1904.959458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1904.963634] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1904.967748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1907.262365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1907.276441] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1915.112920] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 06:27:46 (1778840866) [ 1915.189416] LNet: 106688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.195564] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.201387] LNet: Removed LNI 192.168.201.55@tcp [ 1916.256356] LNet: Removed LNI 192.168.201.55@tcp1 [ 1916.262288] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1916.267082] LNetError: 106688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1916.285701] LNet: Removed LNI 10.1.2.3@tcp99 [ 1916.773031] Key type .llcrypt unregistered [ 1916.774789] Key type ._llcrypt unregistered [ 1921.316312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1921.319976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1923.726834] Key type ._llcrypt registered [ 1923.729242] Key type .llcrypt registered [ 1923.854636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1923.863595] alg: No test for adler32 (adler32-zlib) [ 1924.750979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.800295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1924.821525] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1925.306020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1925.321944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1926.823378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1926.870261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1926.896042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1926.913831] LNetError: 107455:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1926.945036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1926.992647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1927.274712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1927.329671] LNet: 107475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.333835] LNetError: 107475:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.337272] LNet: Removed LNI 192.168.201.55@tcp [ 1927.348925] LNet: Removed LNI 10.1.2.3@tcp [ 1927.830456] Key type .llcrypt unregistered [ 1927.837253] Key type ._llcrypt unregistered [ 1937.239405] Key type ._llcrypt registered [ 1937.243832] Key type .llcrypt registered [ 1938.792969] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 06:28:09 (1778840889) [ 1939.125666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.139303] alg: No test for adler32 (adler32-zlib) [ 1940.082613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.167887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1940.205504] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1940.994946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.55@tcp [ 1941.076562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg155-client@tcp [ 1947.280550] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 06:28:18 (1778840898) [ 1947.378870] LNet: 109003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.393714] LNetError: 109003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.412668] LNet: Removed LNI 192.168.201.55@tcp [ 1947.961575] Key type .llcrypt unregistered [ 1947.963557] Key type ._llcrypt unregistered [ 1952.418218] Key type ._llcrypt registered [ 1952.420733] Key type .llcrypt registered [ 1952.617220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.628362] alg: No test for adler32 (adler32-zlib) [ 1953.492803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1953.511839] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1953.537450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1958.650278] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 06:28:30 (1778840910) [ 1959.691809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1960.938908] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 06:28:32 (1778840912) [ 1962.158118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1963.662396] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 06:28:34 (1778840914) [ 1963.790336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.818483] LNet: 110424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.826507] LNetError: 110424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.853491] LNet: Removed LNI 192.168.201.55@tcp [ 1963.866798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.905439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.922890] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1964.452471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1964.487945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1969.686599] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 06:28:41 (1778840921) [ 1969.752025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.766389] LNet: 111132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.770174] LNetError: 111132:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.786316] LNet: Removed LNI 192.168.201.55@tcp [ 1969.811159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.838254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1969.858992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1970.373543] Key type .llcrypt unregistered [ 1970.376401] Key type ._llcrypt unregistered [ 1978.566474] Key type ._llcrypt registered [ 1978.568432] Key type .llcrypt registered [ 1979.963638] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 06:28:50 (1778840930) [ 1980.344837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.367619] alg: No test for adler32 (adler32-zlib) [ 1981.346166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.425761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.460238] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 1982.179973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1982.243748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 1988.272355] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 06:28:59 (1778840939) [ 1988.537696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.571870] LNet: 112690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.575651] LNetError: 112690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1988.586172] LNet: Removed LNI 192.168.201.55@tcp [ 1988.604968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1988.647805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1988.677839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1994.219520] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 06:29:05 (1778840945) [ 1994.313622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1994.345929] LNet: 113263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.386655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.453031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1994.816952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1994.859035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1994.985692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1995.369156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1995.395745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1995.463588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.485181] LNet: 113303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.507083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.572587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1995.984695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1996.022430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1996.068794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1996.466494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1996.497563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2001.490654] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 06:29:12 (1778840952) [ 2001.593887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2001.622729] LNet: 113898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.670549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.744103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.784606] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2002.364028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2002.376274] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2002.982922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2003.034364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2003.202676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2003.218776] LNetError: 114182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.223663] LNet: Removed LNI 192.168.201.55@tcp [ 2003.235555] LNet: Removed LNI 192.168.201.55@tcp2 [ 2003.258369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.298869] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2003.303918] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2003.307388] LNet: 114188:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2003.520850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2003.661142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.704564] LNet: Removed LNI 192.168.201.55@tcp [ 2004.707959] LNetError: 114225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.711511] LNetError: 114225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2004.723316] LNet: Removed LNI 192.168.201.55@tcp2 [ 2004.747910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.797520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.817579] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2005.327473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2005.342495] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2005.955538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2005.979511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2006.022429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2006.186920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.201365] LNet: 114509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.203759] LNet: 114509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2007.264233] LNet: Removed LNI 192.168.201.55@tcp [ 2007.266710] LNetError: 114509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.277772] LNetError: 114509:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2007.300793] LNet: Removed LNI 192.168.201.55@tcp2 [ 2007.350200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.426698] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2007.432489] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2007.438844] LNet: 114516:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2007.992508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2008.180620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.201761] LNet: Removed LNI 192.168.201.55@tcp [ 2009.254834] LNet: Removed LNI 192.168.201.55@tcp2 [ 2009.285837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.332817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2009.348442] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2009.904186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2009.919604] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2010.630758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2010.678297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2010.844979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2011.872324] LNet: Removed LNI 192.168.201.55@tcp [ 2011.878341] LNetError: 114832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.881679] LNetError: 114832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2011.898786] LNet: Removed LNI 192.168.201.55@tcp2 [ 2011.929195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.979597] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2011.986236] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2011.990352] LNet: 114838:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2012.451921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2012.604532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.664266] LNet: Removed LNI 192.168.201.55@tcp [ 2013.678535] LNet: Removed LNI 192.168.201.55@tcp2 [ 2013.693643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.733845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.759890] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2014.315706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2014.335973] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2015.040276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2015.079723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2015.144613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2015.335772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.351314] LNet: 115160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.356353] LNet: 115160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2016.416172] LNet: Removed LNI 192.168.201.55@tcp [ 2017.451197] LNet: Removed LNI 192.168.201.55@tcp2 [ 2017.465319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.549449] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2017.554056] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2017.558559] LNet: 115167:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.930165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2018.055626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.068961] LNet: Removed LNI 192.168.201.55@tcp [ 2019.112866] LNet: Removed LNI 192.168.201.55@tcp2 [ 2019.143459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.198297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.237513] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2019.807871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2019.831141] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2020.497479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2020.521562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2020.564706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2020.948983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.960264] LNetError: 115500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2020.963160] LNetError: 115500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2020.965839] LNet: Removed LNI 192.168.201.55@tcp [ 2020.971480] LNet: Removed LNI 192.168.201.55@tcp2 [ 2020.981175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.017041] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2021.024387] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2021.404584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2021.694847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.752423] LNet: Removed LNI 192.168.201.55@tcp [ 2022.774435] LNet: Removed LNI 192.168.201.55@tcp2 [ 2022.809046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.891542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.925789] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2023.581933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2023.593610] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2024.350242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2024.381978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2024.405474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2024.466288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2024.843954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.888217] LNet: Removed LNI 192.168.201.55@tcp [ 2026.921500] LNet: Removed LNI 192.168.201.55@tcp2 [ 2027.010273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.141728] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2027.152524] LNet: Added LNI 192.168.201.55@tcp2 [8/256/0/180] [ 2027.162426] LNet: 115866:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2027.171758] LNet: 115866:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2027.872362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2033.746345] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2035.364919] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 06:29:46 (1778840986) [ 2035.569985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.587807] LNet: 116625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.593248] LNet: 116625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2035.601439] LNet: Removed LNI 192.168.201.55@tcp [ 2036.653128] LNet: Removed LNI 192.168.201.55@tcp2 [ 2036.688362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.956306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2036.982964] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2037.517367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2037.533277] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2038.298551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.318723] LNetError: 116898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.322025] LNetError: 116898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2038.325488] LNet: Removed LNI 192.168.201.55@tcp [ 2038.333836] LNet: Removed LNI 10.1.2.3@tcp [ 2038.352770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.406552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2038.429031] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2038.432133] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2039.006675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2039.030920] LNet: Removed LNI 192.168.201.55@tcp [ 2039.043656] LNet: Removed LNI 10.1.2.3@tcp [ 2044.176205] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 06:29:55 (1778840995) [ 2044.241174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.275329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.452927] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2044.564103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2044.566287] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2044.570305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2046.821769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.853362] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2047.330977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2047.344959] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2048.820486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.55@tcp [ 2048.852602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2048.896409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.55@tcp -m PUT -e local_timeout -r 1 [ 2048.938765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2048.979423] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2048.982806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2049.196744] LNetError: 117564:0:(lib-move.c:4710:lnet_parse()) 192.168.201.55@tcp, src 192.168.201.55@tcp: Dropping PUT (error -108 looking up sender) [ 2051.041048] LNet: Removed LNI 192.168.201.55@tcp [ 2051.055843] LNet: Removed LNI 10.1.2.3@tcp [ 2051.396218] Key type .llcrypt unregistered [ 2051.402486] Key type ._llcrypt unregistered [ 2060.514368] Key type ._llcrypt registered [ 2060.516717] Key type .llcrypt registered [ 2061.506266] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 06:30:12 (1778841012) [ 2061.771495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.787471] alg: No test for adler32 (adler32-zlib) [ 2062.685592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.046305] LNet: 118875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.601433] Key type .llcrypt unregistered [ 2063.603878] Key type ._llcrypt unregistered [ 2068.885778] Key type ._llcrypt registered [ 2068.888538] Key type .llcrypt registered [ 2069.029284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2069.045319] alg: No test for adler32 (adler32-zlib) [ 2070.031216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2070.053212] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2076.753800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 2077.006628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2077.501970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.155@tcp [ 2077.727710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2077.751417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2077.770112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2077.795674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2077.821531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2077.843741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2077.867570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2077.893380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2077.910707] LNet: 119325:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2077.931199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2077.965726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2077.995350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2078.037527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2078.077275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2078.108415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2078.658770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2078.861973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2078.885828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2078.927509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2078.954371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2079.008956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2079.059619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2079.104398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2079.146611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2079.187077] LNet: 119443:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2079.211549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2079.250171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2079.264258] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.155@tcp [ 2079.283192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2079.315407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2079.334671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2079.353275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2079.873467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.155@tcp [ 2080.069348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2080.090706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2080.114272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2080.141367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2080.166067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2080.189139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2080.208297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2080.232308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2080.244587] LNet: 119560:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2080.267348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2080.291920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2080.318192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2080.339577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2080.362567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2080.384350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2080.934593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2081.151635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2081.176489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2081.203472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2081.228246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2081.250933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2081.276466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2081.305483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2081.337233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2081.361964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2081.385218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2081.406650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2081.431222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2081.451497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2081.474796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2081.714846] LNet: 119723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.719053] LNetError: 119723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.736317] LNet: Removed LNI 192.168.201.55@tcp [ 2082.039896] Key type .llcrypt unregistered [ 2082.042372] Key type ._llcrypt unregistered [ 2094.010756] Key type ._llcrypt registered [ 2094.012801] Key type .llcrypt registered [ 2095.107493] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 06:30:46 (1778841046) [ 2095.545685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2095.559163] alg: No test for adler32 (adler32-zlib) [ 2096.502183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.869954] LNet: 120611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.344477] Key type .llcrypt unregistered [ 2097.346608] Key type ._llcrypt unregistered [ 2102.373831] Key type ._llcrypt registered [ 2102.377865] Key type .llcrypt registered [ 2102.606701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2102.623750] alg: No test for adler32 (adler32-zlib) [ 2103.509379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2103.525895] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2109.909186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.155@tcp [ 2115.753805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.781937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2115.799481] LNet: Added LNI 192.168.201.55@tcp1 [8/256/0/180] [ 2116.365917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2116.770486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.155@tcp [ 2116.832327] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.155@tcp [ 2116.921093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2116.944728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2116.968514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2116.994435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2117.024561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2117.050916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2117.077787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2117.094314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2117.103490] LNet: 121277:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2117.119426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2117.139947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2117.161280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2117.180403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2117.201733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2117.224770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2117.700655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2117.866512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2117.891318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2117.911761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2117.927851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2117.957079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2117.990840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2118.014660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2118.034934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2118.048906] LNet: 121394:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.062717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2118.083655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2118.109741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2118.132462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2118.156389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2118.190256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2118.653982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.155@tcp [ 2118.841787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2118.862970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2118.884521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2118.905525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2118.930158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2118.949834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2118.970756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2118.991841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2119.014996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2119.040723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2119.062814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2119.086785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2119.111963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2119.136813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2119.550432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2119.711464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2119.735471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2119.770485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2119.801396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2119.833450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2119.867292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2119.906908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2119.936172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2119.955096] LNet: 121629:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.958344] LNet: 121629:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2119.970966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2120.007891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2120.037459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2120.061668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2120.088077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2120.126964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2123.906426] LNet: 121709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.910844] LNetError: 121709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.914929] LNet: Removed LNI 192.168.201.55@tcp [ 2123.927836] LNet: Removed LNI 192.168.201.55@tcp1 [ 2124.219983] Key type .llcrypt unregistered [ 2124.223266] Key type ._llcrypt unregistered [ 2136.062041] Key type ._llcrypt registered [ 2136.063572] Key type .llcrypt registered [ 2136.954447] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 06:31:28 (1778841088) [ 2142.103163] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 06:31:33 (1778841093) [ 2146.686905] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 06:31:38 (1778841098) [ 2152.003731] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 06:31:43 (1778841103) [ 2156.254396] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 06:31:47 (1778841107) [ 2160.604132] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 06:31:52 (1778841112) [ 2164.473755] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 06:31:56 (1778841116) [ 2168.833266] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 06:32:00 (1778841120) [ 2173.266700] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 06:32:04 (1778841124) [ 2177.457634] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 06:32:09 (1778841129) [ 2181.433329] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 06:32:12 (1778841132) [ 2185.850320] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 06:32:17 (1778841137) [ 2190.122248] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 06:32:21 (1778841141) [ 2194.473300] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 06:32:25 (1778841145) [ 2198.625733] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 06:32:30 (1778841150) [ 2202.975703] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 06:32:34 (1778841154) [ 2207.415785] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 06:32:38 (1778841158) [ 2212.676776] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 06:32:44 (1778841164) [ 2212.943269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.956271] alg: No test for adler32 (adler32-zlib) [ 2213.805916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.885191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2213.902789] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2214.614705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.627526] LNet: 132416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.631774] LNetError: 132416:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.648717] LNet: Removed LNI 192.168.201.55@tcp [ 2214.667352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.726389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2214.746516] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2218.566346] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 06:32:50 (1778841170) [ 2222.943818] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 06:32:54 (1778841174) [ 2227.268827] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 06:32:58 (1778841178) [ 2231.529668] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 06:33:03 (1778841183) [ 2235.576665] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 06:33:06 (1778841186) [ 2240.415504] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 06:33:11 (1778841191) [ 2244.859194] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 06:33:16 (1778841196) [ 2248.618900] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 06:33:20 (1778841200) [ 2252.695329] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 06:33:24 (1778841204) [ 2256.033332] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 06:33:27 (1778841207) [ 2259.587731] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 06:33:31 (1778841211) [ 2263.564906] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 06:33:35 (1778841215) [ 2267.967861] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1942 sec ========== 06:33:39 (1778841219) [ 2268.131454] LNet: 139875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.139440] LNetError: 139875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.151708] LNet: Removed LNI 192.168.201.55@tcp [ 2268.521584] Key type .llcrypt unregistered [ 2268.523635] Key type ._llcrypt unregistered [ 2275.465993] Key type ._llcrypt registered [ 2275.468021] Key type .llcrypt registered [ 2275.739662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2275.755375] alg: No test for adler32 (adler32-zlib) [ 2276.700019] Lustre: Lustre: Build Version: 2.17.53_3_g0a4988c [ 2276.913402] LNet: Added LNI 192.168.201.55@tcp [8/256/0/180] [ 2278.560176] Key type lgssc registered [ 2279.264724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2316.180738] Lustre: Mounted lustre-client [ 2319.581603] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2323.688009] LustreError: 142317:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8c5d47ef5000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2323.723100] Lustre: Unmounted lustre-client [ 2355.729254] Key type lgssc unregistered [ 2355.929823] LNet: 142798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.934976] LNetError: 142798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.945880] LNet: Removed LNI 192.168.201.55@tcp [ 2356.354175] Key type .llcrypt unregistered [ 2356.356242] Key type ._llcrypt unregistered