[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 2970185295 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003275] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008295] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011010] pid_max: default: 32768 minimum: 301 [ 0.013128] LSM: Security Framework initializing [ 0.014047] Yama: becoming mindful. [ 0.015031] SELinux: Initializing. [ 0.017046] *** VALIDATE selinux *** [ 0.025287] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029541] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031045] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034064] *** VALIDATE tmpfs *** [ 0.035459] *** VALIDATE proc *** [ 0.036234] *** VALIDATE cgroup *** [ 0.037008] *** VALIDATE cgroup2 *** [ 0.039006] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042027] Spectre V2 : User space: Vulnerable [ 0.043009] Speculative Store Bypass: Vulnerable [ 0.046464] debug: unmapping init [mem 0xffffffffa5a59000-0xffffffffa5a60fff] [ 0.048176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050022] ... version: 2 [ 0.051009] ... bit width: 48 [ 0.052009] ... generic registers: 4 [ 0.053013] ... value mask: 0000ffffffffffff [ 0.054014] ... max period: 00007fffffffffff [ 0.055014] ... fixed-purpose events: 3 [ 0.056010] ... event mask: 000000070000000f [ 0.057279] rcu: Hierarchical SRCU implementation. [ 0.059391] smp: Bringing up secondary CPUs ... [ 0.060566] x86: Booting SMP configuration: [ 0.061023] .... node #0, CPUs: #1 #2 #3 [ 0.068647] smp: Brought up 1 node, 4 CPUs [ 0.070012] smpboot: Max logical packages: 1 [ 0.071011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102016] node 0 deferred pages initialised in 29ms [ 0.110168] devtmpfs: initialized [ 0.111238] x86/mm: Memory block size: 128MB [ 0.113000] gcov: version magic: 0x41383552 [ 0.116331] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121322] pinctrl core: initialized pinctrl subsystem [ 0.124212] [ 0.124836] ************************************************************* [ 0.127014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.130011] ** ** [ 0.133012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.135011] ** ** [ 0.138011] ** This means that this kernel is built to expose internal ** [ 0.141012] ** IOMMU data structures, which may compromise security on ** [ 0.143014] ** your system. ** [ 0.146013] ** ** [ 0.148068] ** If you see this message and you are not debugging the ** [ 0.149013] ** kernel, report this immediately to your vendor! ** [ 0.152011] ** ** [ 0.154011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157011] ************************************************************* [ 0.161159] NET: Registered protocol family 16 [ 0.162585] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.172099] cpuidle: using governor menu [ 0.175013] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.177630] PCI: Using configuration type 1 for base access [ 0.180120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195271] cryptd: max_cpu_qlen set to 1000 [ 0.198233] ACPI: Added _OSI(Module Device) [ 0.199073] ACPI: Added _OSI(Processor Device) [ 0.200016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.202013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.207851] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214646] ACPI: Interpreter enabled [ 0.216054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218013] ACPI: Using IOAPIC for interrupt routing [ 0.220164] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224529] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237942] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.247088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.252689] acpiphp: Slot [2] registered [ 0.254188] acpiphp: Slot [5] registered [ 0.256118] acpiphp: Slot [6] registered [ 0.258174] acpiphp: Slot [3] registered [ 0.260089] acpiphp: Slot [4] registered [ 0.261170] acpiphp: Slot [7] registered [ 0.263089] acpiphp: Slot [8] registered [ 0.264145] acpiphp: Slot [9] registered [ 0.266084] acpiphp: Slot [10] registered [ 0.268109] acpiphp: Slot [11] registered [ 0.269149] acpiphp: Slot [12] registered [ 0.271101] acpiphp: Slot [13] registered [ 0.273116] acpiphp: Slot [14] registered [ 0.274150] acpiphp: Slot [15] registered [ 0.276219] acpiphp: Slot [16] registered [ 0.278077] acpiphp: Slot [17] registered [ 0.279112] acpiphp: Slot [18] registered [ 0.281149] acpiphp: Slot [19] registered [ 0.283099] acpiphp: Slot [20] registered [ 0.284143] acpiphp: Slot [21] registered [ 0.286143] acpiphp: Slot [22] registered [ 0.288146] acpiphp: Slot [23] registered [ 0.289148] acpiphp: Slot [24] registered [ 0.291078] acpiphp: Slot [25] registered [ 0.292127] acpiphp: Slot [26] registered [ 0.294089] acpiphp: Slot [27] registered [ 0.295084] acpiphp: Slot [28] registered [ 0.297099] acpiphp: Slot [29] registered [ 0.298236] acpiphp: Slot [30] registered [ 0.300084] acpiphp: Slot [31] registered [ 0.302064] PCI host bridge to bus 0000:00 [ 0.303082] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.306025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.312018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.315021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.317151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.320050] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.323604] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.332012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.336065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.339017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.342013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.344013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346735] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349905] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.355808] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.359020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.368693] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.372881] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379499] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.436022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.457017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.524171] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.553019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.580017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.657022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.691173] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.701419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704411] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.706391] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.712112] iommu: Default domain type: Passthrough [ 0.713000] SCSI subsystem initialized [ 0.715198] ACPI: bus type USB registered [ 0.716105] usbcore: registered new interface driver usbfs [ 0.718215] usbcore: registered new interface driver hub [ 0.720067] usbcore: registered new device driver usb [ 0.722137] pps_core: LinuxPPS API ver. 1 registered [ 0.724010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.727058] PTP clock support registered [ 0.729106] EDAC MC: Ver: 3.0.0 [ 0.731065] PCI: Using ACPI for IRQ routing [ 0.731910] NetLabel: Initializing [ 0.733104] NetLabel: domain hash size = 128 [ 0.735010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.737148] NetLabel: unlabeled traffic allowed by default [ 0.739328] vgaarb: loaded [ 0.741069] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.742015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.748523] clocksource: Switched to clocksource kvm-clock [ 0.922885] VFS: Disk quotas dquot_6.6.0 [ 0.924555] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.928986] *** VALIDATE ramfs *** [ 0.930181] *** VALIDATE hugetlbfs *** [ 0.932357] pnp: PnP ACPI init [ 0.935058] pnp: PnP ACPI: found 6 devices [ 0.953267] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.956270] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.958218] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.960302] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.962078] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.963727] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.966431] NET: Registered protocol family 2 [ 0.968764] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.973354] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.976813] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.982126] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.985585] TCP: Hash tables configured (established 65536 bind 65536) [ 0.988190] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.991018] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.993667] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.996412] NET: Registered protocol family 1 [ 0.998701] RPC: Registered named UNIX socket transport module. [ 1.000580] RPC: Registered udp transport module. [ 1.002119] RPC: Registered tcp transport module. [ 1.003696] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.005961] NET: Registered protocol family 44 [ 1.007533] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.009536] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.011736] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.013812] PCI: CLS 0 bytes, default 64 [ 1.015778] Unpacking initramfs... [ 4.021314] debug: unmapping init [mem 0xffff8f9d3cc64000-0xffff8f9d3ffcffff] [ 4.025099] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.027525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.030630] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.689073] Initialise system trusted keyrings [ 4.690589] Key type blacklist registered [ 4.719629] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.859531] zbud: loaded [ 4.864708] *** VALIDATE nfs *** [ 4.866046] *** VALIDATE nfs4 *** [ 4.867736] pstore: using deflate compression [ 4.890249] Platform Keyring initialized [ 5.022573] NET: Registered protocol family 38 [ 5.024411] Key type asymmetric registered [ 5.025555] Asymmetric key parser 'x509' registered [ 5.026841] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.030638] io scheduler mq-deadline registered [ 5.032388] io scheduler kyber registered [ 5.034396] io scheduler bfq registered [ 5.036208] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.039139] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.041552] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.043957] ACPI: Power Button [PWRF] [ 5.049587] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.055673] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.247046] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.272534] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.300952] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.305603] Non-volatile memory driver v1.3 [ 5.307407] Linux agpgart interface v0.103 [ 5.569867] virtio_blk virtio1: [vda] 144816 512-byte logical blocks (74.1 MB/70.7 MiB) [ 5.577539] vda: detected capacity change from 0 to 74145792 [ 5.639011] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.646664] vdb: detected capacity change from 0 to 1073741824 [ 5.700658] libphy: Fixed MDIO Bus: probed [ 5.750725] usbcore: registered new interface driver usbserial_generic [ 5.753070] usbserial: USB Serial support registered for generic [ 5.761121] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.768693] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.770896] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.775978] mousedev: PS/2 mouse device common for all mice [ 5.781958] rtc_cmos 00:05: RTC can wake from S4 [ 5.784359] rtc_cmos 00:05: registered as rtc0 [ 5.790831] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.794444] intel_pstate: CPU model not supported [ 5.812260] hid: raw HID events driver (C) Jiri Kosina [ 5.827586] usbcore: registered new interface driver usbhid [ 5.829732] usbhid: USB HID core driver [ 5.840729] drop_monitor: Initializing network drop monitor service [ 5.840954] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.858274] Initializing XFRM netlink socket [ 5.858744] NET: Registered protocol family 10 [ 5.865750] Segment Routing with IPv6 [ 5.881983] NET: Registered protocol family 17 [ 5.908198] mpls_gso: MPLS GSO support [ 5.938643] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.954996] RAS: Correctable Errors collector initialized. [ 5.956749] AVX version of gcm_enc/dec engaged. [ 5.969772] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.969984] AES CTR mode by8 optimization enabled [ 6.649587] sched_clock: Marking stable (6649545966, 0)->(7723762604, -1074216638) [ 6.677320] registered taskstats version 1 [ 6.684960] Loading compiled-in X.509 certificates [ 6.691593] zswap: loaded using pool lzo/zbud [ 6.875283] Key type big_key registered [ 6.954776] Key type encrypted registered [ 6.966669] ima: No TPM chip found, activating TPM-bypass! [ 6.991851] ima: Allocated hash algorithm: sha1 [ 6.993620] ima: No architecture policies found [ 7.012585] evm: Initialising EVM extended attributes: [ 7.023989] evm: security.selinux [ 7.036426] evm: security.ima [ 7.044894] evm: security.capability [ 7.056233] evm: HMAC attrs: 0x1 [ 7.086654] rtc_cmos 00:05: setting system clock to 2026-06-26 13:27:46 UTC (1782480466) [ 7.111649] debug: unmapping init [mem 0xffffffffa6a03000-0xffffffffa6bfffff] [ 7.171696] debug: unmapping init [mem 0xffffffffa5782000-0xffffffffa5a58fff] [ 7.227620] Write protecting the kernel read-only data: 28672k [ 7.265269] debug: unmapping init [mem 0xffffffffa3e03000-0xffffffffa3ffffff] [ 7.288292] debug: unmapping init [mem 0xffffffffa4714000-0xffffffffa47fffff] [ 7.507067] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.539439] systemd[1]: Detected virtualization kvm. [ 7.543429] systemd[1]: Detected architecture x86-64. [ 7.545940] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.617090] systemd[1]: No hostname configured. [ 7.624094] systemd[1]: Set hostname to . [ 7.635679] random: systemd: uninitialized urandom read (16 bytes read) [ 7.637781] systemd[1]: Initializing machine ID from random generator. [ 7.877610] random: ln: uninitialized urandom read (6 bytes read) [ 8.361346] random: systemd: uninitialized urandom read (16 bytes read) [ 8.382101] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.406337] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 8.422921] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ 8.707167] urandom_read: 6 callbacks suppressed [ 8.707178] random: systemd: uninitialized urandom read (16 bytes read) Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.627306] device-mapper: uevent: version 1.0.3 [ 11.665749] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 17.450136] random: fast init done [ 17.512116] virtio_net virtio0 ens2: renamed from eth0 [ 18.390223] scsi host0: ata_piix [ 18.787789] scsi host1: ata_piix [ 18.827481] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 18.848683] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 24.633253] random: crng init done [ 29.974567] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ 33.327512] hrtimer: interrupt took 4182129 ns [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 33.583639] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 38.428329] printk: systemd: 26 output lines suppressed due to ratelimiting [ 39.630608] SELinux: Disabled at runtime. [ 39.982489] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 39.995890] systemd[1]: Detected virtualization kvm. [ 39.999065] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 42.395375] systemd[1]: initrd-switch-root.service: Succeeded. [ 42.412632] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 42.465290] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 42.473839] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 42.481372] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 42.517120] systemd[1]: Starting Journal Service... Starting Journal Service... [ 42.537661] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ 42.873102] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 44.793851] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 46.968629] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 46.974571] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 47.572593] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 47.709562] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 53.690847] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 54.623841] NFS: Registering the id_resolver key type [ 54.636986] Key type id_resolver registered [ 54.644122] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg121-client login: [ 121.799271] libcfs: loading out-of-tree module taints kernel. [ 122.074377] Key type ._llcrypt registered [ 122.075931] Key type .llcrypt registered [ 122.642899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 122.661378] alg: No test for adler32 (adler32-zlib) [ 124.113162] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 124.851610] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 126.727283] Key type lgssc registered [ 128.190808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 265.056228] Lustre: Mounted lustre-client [ 270.357957] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.173832] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing check_logdir /tmp/testlogs/ [ 289.673730] Lustre: DEBUG MARKER: oleg121-client.virtnet: executing yml_node [ 290.784587] Lustre: lustre-OST0000-osc-ffff8f9d8c2ec000: disconnect after 24s idle [ 294.447766] Lustre: DEBUG MARKER: Client: 2.17.54.98 [ 297.286429] Lustre: DEBUG MARKER: MDS: 2.17.54.98 [ 300.032785] Lustre: DEBUG MARKER: OSS: 2.17.54.98 [ 301.829173] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jun 26 09:32:39 EDT 2026 [ 319.551825] Lustre: Unmounted lustre-client [ 340.374053] Key type lgssc unregistered [ 340.745341] LNet: 8387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.756817] LNetError: 8387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.773442] LNet: Removed LNI 192.168.201.21@tcp [ 341.579597] Key type .llcrypt unregistered [ 341.587440] Key type ._llcrypt unregistered [ 373.257024] Key type ._llcrypt registered [ 373.258231] Key type .llcrypt registered [ 373.329121] LNet: 8969:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 373.329121] mask usage: [+|-] ... [ 373.585322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 373.606599] alg: No test for adler32 (adler32-zlib) [ 374.753810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 374.882281] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 374.928884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 380.153502] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 380.521443] LNet: 9275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.536919] LNetError: 9275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.556315] LNet: Removed LNI 192.168.201.21@tcp [ 381.528494] Key type .llcrypt unregistered [ 381.533367] Key type ._llcrypt unregistered [ 391.220482] Key type ._llcrypt registered [ 391.226895] Key type .llcrypt registered [ 392.856580] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:34:10 (1782480850) [ 393.020265] LNet: 9642:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 393.020265] mask usage: [+|-] ... [ 393.205695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 393.226558] alg: No test for adler32 (adler32-zlib) [ 394.192244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.255477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.275277] LNet: 9686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.289656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.337572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 394.351776] LNet: 9693:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 403.156965] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:34:20 (1782480860) [ 403.266881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.317238] LNet: 10267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.354559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.517296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 412.870764] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:34:30 (1782480870) [ 412.980220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.044055] LNet: 10854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.091624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.273522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 423.039538] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:34:40 (1782480880) [ 423.190121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.234809] LNet: 11440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.267888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.407529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 432.737949] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:34:50 (1782480890) [ 432.886955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.926373] LNet: 12025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.973695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.104496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 433.602842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 441.944486] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:34:59 (1782480899) [ 442.080852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.120355] LNet: 12616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.182850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.381555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 443.221978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 444.160300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 452.982333] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:35:10 (1782480910) [ 453.256876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.320050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.398702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 463.405527] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:35:20 (1782480920) [ 464.018626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.063778] LNet: 13804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.066882] LNet: 13804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 464.098544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.201310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 464.266520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 464.992461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 465.025907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 465.492172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 465.518500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 465.800456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 465.830433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 466.148379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 466.178204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 466.734379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 466.782861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 467.512022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 467.582899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 468.018215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 468.080487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 468.789815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 468.834599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 469.555656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 469.651414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 470.163395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 470.220841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 470.909761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 470.952453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 471.554236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 471.621115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 479.033273] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:35:36 (1782480936) [ 479.222915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.268160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.331227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 479.376983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 489.659688] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:35:47 (1782480947) [ 490.615865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.679499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.842902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 490.932424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 500.862419] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:35:58 (1782480958) [ 501.070590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.124297] LNet: 15662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 501.128777] LNet: 15662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 501.146671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.213907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 501.253401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 510.725962] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:36:08 (1782480968) [ 511.185963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.244511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.350257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 511.398507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 519.094659] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:36:16 (1782480976) [ 519.215663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.310496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.465996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 519.531319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 527.970531] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:36:25 (1782480985) [ 528.150790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.234583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.330347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 528.366961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 537.203436] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:36:34 (1782480994) [ 537.323300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.373759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.535429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.985887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 546.198525] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:36:43 (1782481003) [ 546.282775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.357936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.681567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.041686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 555.584637] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:36:53 (1782481013) [ 555.895179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.932792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.046632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 556.286605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 564.706384] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:37:01 (1782481021) [ 564.994709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.085100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.597530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.910103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 574.865800] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:37:12 (1782481032) [ 575.074975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.124695] LNet: 20491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.127962] LNet: 20491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 575.161209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.243148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 575.340207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 584.396927] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:37:22 (1782481042) [ 584.652826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.721948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.847549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.945682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 593.515608] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:37:31 (1782481051) [ 593.813038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.858331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.977445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 602.871224] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:37:40 (1782481060) [ 603.474540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.546778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.685316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 612.917676] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:37:50 (1782481070) [ 613.142795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.245471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.378760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 614.129755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 622.059158] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:37:59 (1782481079) [ 622.304522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.342744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.426790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 622.467668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 629.209141] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:38:07 (1782481087) [ 629.481330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.511886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.570171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 629.599717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 637.380396] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:38:15 (1782481095) [ 637.736808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.803799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.944312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 637.978779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 645.649133] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:38:23 (1782481103) [ 645.757235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.847834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.968406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 646.025616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 653.996434] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:38:31 (1782481111) [ 654.079280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.139986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.176368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 654.231231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 654.260727] LNetError: 25815:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 654.301637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 654.331453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 660.821570] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:38:38 (1782481118) [ 661.160981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.201627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.283515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hx6eFeVjSR [ 661.374979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hx6eFeVjSR [ 661.433252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Hx6eFeVjSR [ 668.416432] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:38:46 (1782481126) [ 668.582774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.685990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.818037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 668.869829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 668.963286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 676.598953] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:38:54 (1782481134) [ 677.024831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.090928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.176734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 680.839283] LNet: Message forwarding will be enabled in 2s [ 683.938058] LNet: Message forwarding enabled [ 684.447562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 684.504364] LNet: Message forwarding will stop in 2s to 6s [ 689.240892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 689.789433] LNet: Message forwarding disabled [ 691.470780] LNet: Message forwarding will be enabled in 2s [ 694.944538] LNet: Message forwarding enabled [ 699.518455] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:39:17 (1782481157) [ 699.601073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.711382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.810508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 707.173750] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:39:25 (1782481165) [ 707.377870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.396988] LNet: 28845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.401921] LNet: 28845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 707.425560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.483902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 707.509449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 707.531572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 707.562592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 707.601045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 707.646380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 707.680577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 707.722656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 707.767133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 707.801024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 707.873210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 707.905344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 707.949965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 707.998134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 708.068447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 708.144726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 708.204414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 708.271085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 708.346078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 708.393940] LNetError: 28888:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 715.331155] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:39:33 (1782481173) [ 715.612629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.689836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.813601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 723.046436] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:39:40 (1782481180) [ 725.331985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg121-server [ 727.393129] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:39:45 (1782481185) [ 727.483309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.532729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.598234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.624586] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 729.410323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[22-26]@tcp [ 729.438163] LNet: 30605:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 729.507118] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 729.519636] LNetError: 30391:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.23@tcp has gone from up to down [ 730.475902] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 730.496632] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 731.496658] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 731.506670] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 733.566890] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 733.587322] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 737.643228] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.24@tcp is being used as a gateway but routing feature is not turned on [ 737.654547] LNetError: 30391:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 738.066473] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:39:55 (1782481195) [ 740.412480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg121-server [ 743.024342] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:40:00 (1782481200) [ 743.199542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.264315] LNetError: 31534:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.318938] LNet: Removed LNI 192.168.201.21@tcp [ 743.400306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.537806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.581646] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 744.283310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[22-27/2]@tcp [ 744.301292] LNet: 31658:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 744.306905] LNet: 31658:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 744.334768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[22-27/2]@tcp [ 744.340393] LNetError: 31545:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.22@tcp has gone from up to down [ 744.371294] LNetError: 31545:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 751.912094] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:40:09 (1782481209) [ 752.260128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.278272] LNetError: 32251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.325664] LNet: Removed LNI 192.168.201.21@tcp [ 753.365977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.473601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 753.557954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.605479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.676590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 753.742561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.779800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.876969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 753.962944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.014832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.085707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 754.187248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.229290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.312564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 754.391307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.440238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.496724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 754.566266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.601663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.662548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 754.730909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.760518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.834716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 754.943138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.991939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.058686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 755.184701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.252857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.329646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 755.400388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.431755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.485461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 755.536297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.553842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.599881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 762.611555] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:40:20 (1782481220) [ 764.637644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg121-server [ 766.614941] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:40:24 (1782481224) [ 768.626490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg121-server [ 770.658393] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:40:28 (1782481228) [ 771.360805] Key type .llcrypt unregistered [ 771.363899] Key type ._llcrypt unregistered [ 778.797563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.058275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.070389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.076063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.727621] Key type ._llcrypt registered [ 781.730801] Key type .llcrypt registered [ 781.766679] LNet: 33931:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 781.766679] mask usage: [+|-] ... [ 781.891919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.906748] alg: No test for adler32 (adler32-zlib) [ 782.924234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.031911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 783.078313] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 783.760953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.779956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.562712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 785.590578] LNet: Removed LNI 10.1.2.3@tcp [ 785.733995] LNet: 34373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.743865] LNetError: 34373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.767197] LNet: Removed LNI 192.168.201.21@tcp [ 786.314241] Key type .llcrypt unregistered [ 786.316654] Key type ._llcrypt unregistered [ 798.337449] Key type ._llcrypt registered [ 798.339466] Key type .llcrypt registered [ 799.516450] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:40:57 (1782481257) [ 800.129528] Key type .llcrypt unregistered [ 800.131050] Key type ._llcrypt unregistered [ 806.211361] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.411359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.414366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.416951] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.062725] Key type ._llcrypt registered [ 809.066327] Key type .llcrypt registered [ 809.125657] LNet: 35409:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 809.125657] mask usage: [+|-] ... [ 809.417382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.447197] alg: No test for adler32 (adler32-zlib) [ 810.442896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.484066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.504054] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 811.412371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.430742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 814.140402] LNet: 35868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.151286] LNetError: 35868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 814.161743] LNet: Removed LNI 10.1.2.3@tcp [ 814.169327] LNet: Removed LNI 192.168.201.21@tcp [ 814.791451] Key type .llcrypt unregistered [ 814.796198] Key type ._llcrypt unregistered [ 825.255783] Key type ._llcrypt registered [ 825.258741] Key type .llcrypt registered [ 826.450942] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:41:24 (1782481284) [ 826.970079] Key type .llcrypt unregistered [ 826.974974] Key type ._llcrypt unregistered [ 832.795517] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.854914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.859333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.862536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.361724] Key type ._llcrypt registered [ 835.366276] Key type .llcrypt registered [ 835.395731] LNet: 36905:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 835.395731] mask usage: [+|-] ... [ 835.516434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.527924] alg: No test for adler32 (adler32-zlib) [ 836.466376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.565339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 836.598471] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 838.473571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.488638] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.390580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 840.412070] LNet: Removed LNI 10.1.2.3@tcp [ 840.426323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 840.482307] LNet: Removed LNI 10.1.2.31@tcp [ 840.530882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.597170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 841.306452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 841.330801] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 842.162406] LNet: 37788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.166496] LNetError: 37788:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 842.172971] LNet: Removed LNI 10.1.2.31@tcp [ 842.181878] LNet: Removed LNI 10.1.2.3@tcp [ 842.701982] Key type .llcrypt unregistered [ 842.705805] Key type ._llcrypt unregistered [ 853.621711] Key type ._llcrypt registered [ 853.624960] Key type .llcrypt registered [ 854.928684] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:41:52 (1782481312) [ 855.275125] LNet: 38581:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 855.275125] mask usage: [+|-] ... [ 855.470696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.491177] alg: No test for adler32 (adler32-zlib) [ 856.469416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.498291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.510407] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 857.630961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.660720] LNet: 38741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.673991] LNetError: 38741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.690389] LNet: Removed LNI 192.168.201.21@tcp [ 857.715992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.776806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.21@tcp [ 857.821247] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 858.759599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.21@tcp [ 858.797342] LNet: Removed LNI 192.168.201.21@tcp [ 858.900562] LNet: 38876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.771830] Key type .llcrypt unregistered [ 859.774960] Key type ._llcrypt unregistered [ 870.935871] Key type ._llcrypt registered [ 870.944043] Key type .llcrypt registered [ 872.285635] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:42:10 (1782481330) [ 874.204760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg121-server [ 875.801521] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:42:13 (1782481333) [ 876.356431] Key type .llcrypt unregistered [ 876.358895] Key type ._llcrypt unregistered [ 883.111046] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.298302] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 883.307623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 883.316971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.974524] Key type ._llcrypt registered [ 885.976561] Key type .llcrypt registered [ 886.016082] LNet: 40253:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 886.016082] mask usage: [+|-] ... [ 886.213227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.226390] alg: No test for adler32 (adler32-zlib) [ 887.112337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.161684] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 887.885219] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 889.885896] LNet: 40713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.890792] LNetError: 40713:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.897685] LNet: Removed LNI 192.168.201.21@tcp1 [ 889.914177] LNet: Removed LNI 10.1.2.3@tcp2 [ 890.496380] Key type .llcrypt unregistered [ 890.499375] Key type ._llcrypt unregistered [ 900.692895] Key type ._llcrypt registered [ 900.695141] Key type .llcrypt registered [ 901.938193] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:42:39 (1782481359) [ 902.162119] LNet: 41518:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 902.162119] mask usage: [+|-] ... [ 902.301862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.316697] alg: No test for adler32 (adler32-zlib) [ 903.268214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.326893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.355846] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 904.254527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.275294] LNet: 41684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.282786] LNetError: 41684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.301715] LNet: Removed LNI 192.168.201.21@tcp [ 904.314307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.374207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 904.394212] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 910.302193] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:42:48 (1782481368) [ 910.411405] LNet: 42267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.424137] LNetError: 42267:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.443694] LNet: Removed LNI 192.168.201.21@tcp [ 911.114422] Key type .llcrypt unregistered [ 911.118153] Key type ._llcrypt unregistered [ 917.152537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.248168] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.251323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.255362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.930428] Key type ._llcrypt registered [ 919.932081] Key type .llcrypt registered [ 919.972710] LNet: 42553:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 919.972710] mask usage: [+|-] ... [ 920.195687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.218812] alg: No test for adler32 (adler32-zlib) [ 921.176846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.246337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.274315] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 921.914787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 921.934625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.254177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.270891] LNet: 43019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.277952] LNetError: 43019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.283362] LNet: Removed LNI 10.1.2.3@tcp [ 924.292258] LNet: Removed LNI 192.168.201.21@tcp [ 924.306621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.364896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 924.403232] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 924.408707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 924.858776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.879914] LNet: 43047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.885893] LNetError: 43047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 924.895389] LNetError: 43047:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 924.906344] LNet: Removed LNI 10.1.2.3@tcp [ 924.927109] LNet: Removed LNI 192.168.201.21@tcp [ 924.964022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.057137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 925.082413] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 925.086661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.431579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 925.455763] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 926.421318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.435241] LNet: 43210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.441827] LNetError: 43210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.447818] LNetError: 43210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 926.454497] LNet: Removed LNI 10.1.2.3@tcp [ 926.456181] LNet: Removed LNI 192.168.201.21@tcp [ 926.474182] LNet: Removed LNI 192.168.201.21@tcp2 [ 926.495661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.547402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 926.568513] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 926.572915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 926.577773] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 934.420527] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:43:12 (1782481392) [ 934.635618] LNet: 43804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.646615] LNetError: 43804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 934.651665] LNetError: 43804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 934.663389] LNet: Removed LNI 10.1.2.3@tcp [ 934.666110] LNet: Removed LNI 192.168.201.21@tcp [ 934.691648] LNet: Removed LNI 192.168.201.21@tcp2 [ 935.298341] Key type .llcrypt unregistered [ 935.299885] Key type ._llcrypt unregistered [ 941.257759] Key type ._llcrypt registered [ 941.260494] Key type .llcrypt registered [ 941.300698] LNet: 44033:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 941.300698] mask usage: [+|-] ... [ 941.519528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.550644] alg: No test for adler32 (adler32-zlib) [ 942.554622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.609604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 942.641618] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 943.290477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.308978] LNet: 44203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.315564] LNetError: 44203:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.329620] LNet: Removed LNI 192.168.201.21@tcp [ 943.344167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.389984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 943.407081] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 943.991329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.020494] LNet: 44233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.026744] LNetError: 44233:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.041770] LNet: Removed LNI 192.168.201.21@tcp [ 944.058572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.135801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.166228] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 944.671310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.704354] LNet: Removed LNI 192.168.201.21@tcp [ 944.719931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.780960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.803242] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 945.204462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 945.346869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 945.357094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 945.366404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.751570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.779746] LNet: 44341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.785943] LNet: 44341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 947.792511] LNetError: 44341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.795712] LNetError: 44341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 947.810677] LNet: Removed LNI 192.168.201.21@tcp [ 947.824207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.875818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 947.894431] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 953.615454] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:43:31 (1782481411) [ 953.723061] LNet: 44930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.731916] LNetError: 44930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 953.763895] LNet: Removed LNI 192.168.201.21@tcp [ 954.326205] Key type .llcrypt unregistered [ 954.327657] Key type ._llcrypt unregistered [ 959.851855] Key type ._llcrypt registered [ 959.853740] Key type .llcrypt registered [ 959.892565] LNet: 45163:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 959.892565] mask usage: [+|-] ... [ 960.005948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.019372] alg: No test for adler32 (adler32-zlib) [ 960.980728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.072448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.099678] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 961.855851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.873690] LNet: 45336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.878320] LNetError: 45336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.890867] LNet: Removed LNI 192.168.201.21@tcp [ 961.910682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.986537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 962.017278] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 967.238635] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:43:45 (1782481425) [ 967.507436] LNet: 45928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.511279] LNetError: 45928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.523676] LNet: Removed LNI 192.168.201.21@tcp [ 968.077560] Key type .llcrypt unregistered [ 968.080389] Key type ._llcrypt unregistered [ 973.906448] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.011258] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.018861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.022640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.702561] Key type ._llcrypt registered [ 976.705582] Key type .llcrypt registered [ 976.743462] LNet: 46205:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 976.743462] mask usage: [+|-] ... [ 976.870594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.890985] alg: No test for adler32 (adler32-zlib) [ 977.848671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.927441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.962447] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 978.656659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.675946] LNet: 46379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.687486] LNetError: 46379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.696621] LNet: Removed LNI 192.168.201.21@tcp [ 978.718375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.770412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 978.791636] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 984.348810] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:44:02 (1782481442) [ 984.549978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.586841] LNet: 46976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.590661] LNetError: 46976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.602911] LNet: Removed LNI 192.168.201.21@tcp [ 984.620054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.714862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 990.214829] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:44:08 (1782481448) [ 990.369841] LNet: 47554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.013370] Key type .llcrypt unregistered [ 991.014649] Key type ._llcrypt unregistered [ 996.730336] Key type ._llcrypt registered [ 996.732248] Key type .llcrypt registered [ 996.765972] LNet: 47784:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 996.765972] mask usage: [+|-] ... [ 996.910701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.929490] alg: No test for adler32 (adler32-zlib) [ 997.878266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 997.925238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 997.943987] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 998.580950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.600620] LNet: 47956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.608117] LNetError: 47956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.625249] LNet: Removed LNI 192.168.201.21@tcp [ 998.644867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.720580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 998.792389] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1005.480347] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:44:23 (1782481463) [ 1005.639996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.704397] LNet: 48549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.714092] LNetError: 48549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.722201] LNet: Removed LNI 192.168.201.21@tcp [ 1005.762575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.799988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1011.249847] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:44:29 (1782481469) [ 1012.920042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1015.081035] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:44:32 (1782481472) [ 1016.840468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1018.611297] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:44:36 (1782481476) [ 1020.103516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1021.428294] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:44:39 (1782481479) [ 1022.966648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1024.302336] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:44:42 (1782481482) [ 1025.755875] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1026.966932] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:44:45 (1782481485) [ 1027.185492] LNet: 50598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.814808] Key type .llcrypt unregistered [ 1027.816680] Key type ._llcrypt unregistered [ 1033.156518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.270084] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.272773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.275618] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.758564] Key type ._llcrypt registered [ 1035.770400] Key type .llcrypt registered [ 1035.814762] LNet: 50882:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1035.814762] mask usage: [+|-] ... [ 1035.968187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.986426] alg: No test for adler32 (adler32-zlib) [ 1036.942549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.029552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.093117] LNet: Added LNI 192.168.201.21@tcp [8/512/0/180] [ 1037.103341] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1037.111351] LNet: 50920:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.242724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.277766] LNet: 50939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.300849] LNetError: 50939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.310115] LNet: Removed LNI 10.1.2.3@tcp [ 1037.313091] LNetError: 50939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.333777] LNet: Removed LNI 192.168.201.21@tcp [ 1037.368416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.431987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.459059] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1037.562568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.594132] LNet: 50964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.599060] LNetError: 50964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.616547] LNet: Removed LNI 192.168.201.21@tcp [ 1037.634101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.724629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.739933] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1037.744123] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.746734] LNet: 50972:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.873192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.911541] LNet: 50990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.916256] LNetError: 50990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.928365] LNet: Removed LNI 10.1.2.3@tcp [ 1037.931703] LNetError: 50990:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.944733] LNet: Removed LNI 192.168.201.21@tcp [ 1037.967244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.031577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.056109] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1038.061791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1038.066247] LNet: 50999:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.207488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.228762] LNet: 51025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.241513] LNetError: 51025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.249622] LNet: Removed LNI 10.1.2.3@tcp [ 1038.251150] LNetError: 51025:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.273368] LNet: Removed LNI 192.168.201.21@tcp [ 1038.307321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.387906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.424672] LNet: Added LNI 192.168.201.21@tcp [8/256/0/182] [ 1038.430755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.546548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1038.572271] LNet: 51053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.580209] LNetError: 51053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.586351] LNet: Removed LNI 10.1.2.3@tcp [ 1038.588495] LNetError: 51053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.611280] LNet: Removed LNI 192.168.201.21@tcp [ 1038.652414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.743772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.767883] LNet: Added LNI 192.168.201.21@tcp [8/256/0/182] [ 1038.772813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.776509] LNet: 51061:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.851732] LNet: Removed LNI 10.1.2.3@tcp [ 1038.860203] LNet: Removed LNI 192.168.201.21@tcp [ 1039.410331] Key type .llcrypt unregistered [ 1039.412546] Key type ._llcrypt unregistered [ 1050.764479] Key type ._llcrypt registered [ 1050.766203] Key type .llcrypt registered [ 1051.872046] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:45:10 (1782481510) [ 1052.109758] LNet: 51876:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1052.109758] mask usage: [+|-] ... [ 1052.234299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1052.256692] alg: No test for adler32 (adler32-zlib) [ 1053.157675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.193330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1053.211041] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1053.793960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.21 [ 1053.807431] LNet: 52037:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1053.813860] LNet: Removed LNI 192.168.201.21@tcp [ 1059.778965] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:45:17 (1782481517) [ 1059.992422] LNet: 52607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.998610] LNetError: 52607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.010458] LNet: Removed LNI 192.168.201.21@tcp [ 1060.591350] Key type .llcrypt unregistered [ 1060.597238] Key type ._llcrypt unregistered [ 1065.735480] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1065.916397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1068.582868] Key type ._llcrypt registered [ 1068.585183] Key type .llcrypt registered [ 1068.608706] LNet: 52890:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1068.608706] mask usage: [+|-] ... [ 1068.718756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.734270] alg: No test for adler32 (adler32-zlib) [ 1069.909936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1074.990205] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:45:33 (1782481533) [ 1075.167508] LNet: 53522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.181684] LNetError: 53522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1075.193768] LNet: Removed LNI 10.1.2.3@tcp [ 1075.717443] Key type .llcrypt unregistered [ 1075.718964] Key type ._llcrypt unregistered [ 1081.444405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1081.669979] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1081.672662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1081.676255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1084.305721] Key type ._llcrypt registered [ 1084.308975] Key type .llcrypt registered [ 1084.344774] LNet: 53796:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1084.344774] mask usage: [+|-] ... [ 1084.458683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.480620] alg: No test for adler32 (adler32-zlib) [ 1085.461492] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1090.673304] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:45:49 (1782481549) [ 1092.234984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1093.424793] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:45:51 (1782481551) [ 1093.509993] LNet: 54700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1093.516720] LNetError: 54700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1093.537192] LNet: Removed LNI 10.1.2.3@tcp [ 1094.015597] Key type .llcrypt unregistered [ 1094.017971] Key type ._llcrypt unregistered [ 1098.853568] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.100474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1099.107092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1099.115805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.958502] Key type ._llcrypt registered [ 1101.961188] Key type .llcrypt registered [ 1102.011769] LNet: 54980:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1102.011769] mask usage: [+|-] ... [ 1102.217823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1102.253780] alg: No test for adler32 (adler32-zlib) [ 1103.234846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.298355] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1110.151564] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:46:08 (1782481568) [ 1110.517802] LNet: 55935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.522951] LNetError: 55935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1110.539194] LNet: Removed LNI 10.1.2.3@tcp [ 1111.053543] Key type .llcrypt unregistered [ 1111.054725] Key type ._llcrypt unregistered [ 1116.267160] Key type ._llcrypt registered [ 1116.269568] Key type .llcrypt registered [ 1116.308287] LNet: 56158:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1116.308287] mask usage: [+|-] ... [ 1116.474488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.502084] alg: No test for adler32 (adler32-zlib) [ 1117.419361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1117.462268] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1124.479653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1125.089252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_interrupt -r 1 [ 1125.116778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_interrupt -r 1 [ 1125.140294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1125.148971] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1125.348738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1125.376642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1125.404935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.427686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.450590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.474618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.512503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.542820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.558808] LNet: 56372:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.575735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.607861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.638304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.664577] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1125.692402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.721183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.749118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.782346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.310326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_dropped -r 1 [ 1126.349660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_dropped -r 1 [ 1126.393759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1126.422307] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1126.666709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.702736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1126.732756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.753924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.777941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.803370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.845374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.879772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.913617] LNet: 56481:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.937815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.971624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.005967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.036496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.079895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.125887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.182650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.571334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_aborted -r 1 [ 1127.609927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_aborted -r 1 [ 1127.638682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1127.665909] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1127.674973] LNet: Skipped 1 previous similar message [ 1127.905600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.944744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1127.975592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.997356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.025358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.053761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.086682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.119667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.134759] LNet: 56591:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.150658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.178818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.206876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.239968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.269389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.297759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.322995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.711211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_no_route -r 1 [ 1128.742847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_no_route -r 1 [ 1128.777679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1128.998387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.024697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1129.051517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.076899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.102347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.131529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.153407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.175872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.202620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.225827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.251273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.273648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.296548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.323253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.347467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.788788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_timeout -r 1 [ 1129.823404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_timeout -r 1 [ 1129.850558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1129.865517] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1129.870561] LNet: Skipped 1 previous similar message [ 1130.036972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.056988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1130.083904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.116886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.139525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.179660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.208089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.250094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.277977] LNet: 56809:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.282340] LNet: 56809:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1130.304500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.339253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.368426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.394705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.424665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.453996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.501698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1130.976337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_error -r 1 [ 1131.011476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_error -r 1 [ 1131.042688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1131.231902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.257867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1131.286597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.317317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.349868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.373805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.406787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.429494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.453551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.478465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.501720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.526344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.560352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.598442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.631201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1131.865895] LNet: 56959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.871411] LNetError: 56959:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.885967] LNet: Removed LNI 192.168.201.21@tcp [ 1132.268440] Key type .llcrypt unregistered [ 1132.270782] Key type ._llcrypt unregistered [ 1147.003231] Key type ._llcrypt registered [ 1147.005882] Key type .llcrypt registered [ 1148.007072] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:46:46 (1782481606) [ 1148.712404] Key type .llcrypt unregistered [ 1148.719218] Key type ._llcrypt unregistered [ 1153.940606] Key type ._llcrypt registered [ 1153.942092] Key type .llcrypt registered [ 1153.968910] LNet: 58007:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1153.968910] mask usage: [+|-] ... [ 1154.131439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.146173] alg: No test for adler32 (adler32-zlib) [ 1155.092292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1155.129933] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1161.226343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1169.120577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.190906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1169.210580] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1170.508566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_interrupt -r 1 [ 1170.557404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_interrupt -r 1 [ 1170.585526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_interrupt -r 1 [ 1170.619917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_interrupt -r 1 [ 1170.645836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_interrupt -r 1 [ 1170.673152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_interrupt -r 1 [ 1170.701769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_interrupt -r 1 [ 1170.727918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_interrupt -r 1 [ 1170.761198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1170.778118] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1170.783790] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1170.987840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.024238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.048946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.079173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.111453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.140734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.166855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.192838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.208732] LNet: 58436:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.223983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.254472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.284474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.314737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.346540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.379665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.415752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.782499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_dropped -r 1 [ 1171.814590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_dropped -r 1 [ 1171.846690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_dropped -r 1 [ 1171.878819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_dropped -r 1 [ 1171.909391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_dropped -r 1 [ 1171.942175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_dropped -r 1 [ 1171.993402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_dropped -r 1 [ 1172.016203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_dropped -r 1 [ 1172.047685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1172.071793] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1172.076472] LNet: Skipped 2 previous similar messages [ 1172.259941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.277860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1172.303840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.330942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.352796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.374703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.393259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.409186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.417238] LNet: 58557:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.432833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.474344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.521163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.561834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.604534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.635250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.657270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1173.160941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_aborted -r 1 [ 1173.196675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_aborted -r 1 [ 1173.235372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_aborted -r 1 [ 1173.286967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_aborted -r 1 [ 1173.320761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_aborted -r 1 [ 1173.367921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_aborted -r 1 [ 1173.407328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_aborted -r 1 [ 1173.457613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_aborted -r 1 [ 1173.505311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1173.528803] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1173.534765] LNet: Skipped 2 previous similar messages [ 1173.724544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.751793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1173.781559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.806219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.828253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.850201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.875778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.900841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.912731] LNet: 58678:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.928065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.951954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.982515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.006832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.032560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.057489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.083752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.478325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_no_route -r 1 [ 1174.515158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_no_route -r 1 [ 1174.546103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_no_route -r 1 [ 1174.578607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_no_route -r 1 [ 1174.605544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_no_route -r 1 [ 1174.641251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_no_route -r 1 [ 1174.681777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_no_route -r 1 [ 1174.708932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_no_route -r 1 [ 1174.742689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1174.955792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.987715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1175.015709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.035486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.068908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.103734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.131910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.173230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.199994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.248171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.277526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.323760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.375660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.414368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.449473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.875338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_timeout -r 1 [ 1175.906438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_timeout -r 1 [ 1175.964142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_timeout -r 1 [ 1175.992124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_timeout -r 1 [ 1176.020041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_timeout -r 1 [ 1176.062718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_timeout -r 1 [ 1176.106612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_timeout -r 1 [ 1176.146319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_timeout -r 1 [ 1176.191423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1176.209355] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1176.213681] LNet: Skipped 5 previous similar messages [ 1176.418388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.448788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.476132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.495684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.523939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.550582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.581857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.617305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.636581] LNet: 58921:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.640960] LNet: 58921:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1176.656950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.685277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.709759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.731854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.757542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.779737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.808980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.218494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_error -r 1 [ 1177.245517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_error -r 1 [ 1177.274635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_error -r 1 [ 1177.301657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_error -r 1 [ 1177.336168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_error -r 1 [ 1177.366796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_error -r 1 [ 1177.403389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_error -r 1 [ 1177.447120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_error -r 1 [ 1177.478787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1177.637679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.663456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1177.691257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.728677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.785415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.823369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.872722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.911764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.959648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.999382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.035682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.060731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.087910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.124434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.155862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1182.236629] LNet: 59117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.240891] LNetError: 59117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.247234] LNet: Removed LNI 192.168.201.21@tcp [ 1182.263843] LNet: Removed LNI 192.168.201.21@tcp1 [ 1182.573692] Key type .llcrypt unregistered [ 1182.575543] Key type ._llcrypt unregistered [ 1195.757733] Key type ._llcrypt registered [ 1195.760556] Key type .llcrypt registered [ 1196.835199] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:47:35 (1782481655) [ 1197.575506] Key type .llcrypt unregistered [ 1197.578255] Key type ._llcrypt unregistered [ 1201.919968] Key type ._llcrypt registered [ 1201.922186] Key type .llcrypt registered [ 1201.950203] LNet: 60151:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1201.950203] mask usage: [+|-] ... [ 1202.088334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.105860] alg: No test for adler32 (adler32-zlib) [ 1202.982648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.003911] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1209.323246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1209.853949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_dropped -r 1 [ 1209.883826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_dropped -r 1 [ 1209.907394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1209.926576] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1210.136839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.173844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1210.220579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.247952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.276052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.315763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.342056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.357525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.369869] LNet: 60366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.389992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.420231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.447346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.476613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.505159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.533406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.559404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.939384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_error -r 1 [ 1210.964637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_error -r 1 [ 1210.988480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1211.002866] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1211.113817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.137592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1211.163856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.186395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.211029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.239546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.286681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.308612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.331482] LNet: 60475:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.370556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.417548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.479695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.537954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.563152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.616603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.657575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.151837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_timeout -r 1 [ 1212.181147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_timeout -r 1 [ 1212.222791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1212.239551] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1212.411641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.431861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.465826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.491928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.509136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.529453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.549754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.572519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.590569] LNet: 60584:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.607477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.637086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.661876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.694576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.737141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.768728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.797677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.012436] LNet: 60625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.021428] LNetError: 60625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.076130] LNet: Removed LNI 192.168.201.21@tcp [ 1214.480623] Key type .llcrypt unregistered [ 1214.487148] Key type ._llcrypt unregistered [ 1227.717553] Key type ._llcrypt registered [ 1227.719364] Key type .llcrypt registered [ 1228.711986] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:48:07 (1782481687) [ 1229.501577] Key type .llcrypt unregistered [ 1229.504962] Key type ._llcrypt unregistered [ 1235.228090] Key type ._llcrypt registered [ 1235.231765] Key type .llcrypt registered [ 1235.296349] LNet: 61661:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1235.296349] mask usage: [+|-] ... [ 1235.463365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.480482] alg: No test for adler32 (adler32-zlib) [ 1236.434389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1236.468671] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1243.390338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1251.568451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.611619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1251.659701] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1252.772304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_dropped -r 1 [ 1252.800996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e remote_dropped -r 1 [ 1252.823691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_dropped -r 1 [ 1252.844221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e remote_dropped -r 1 [ 1252.877360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e remote_dropped -r 1 [ 1252.907400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e remote_dropped -r 1 [ 1252.935697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e remote_dropped -r 1 [ 1252.970503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e remote_dropped -r 1 [ 1252.998614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1253.011532] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1253.015547] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.121@tcp [ 1253.226204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1253.259290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1253.293645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1253.324293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1253.369781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1253.409562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1253.459459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1253.483700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1253.499676] LNet: 62090:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1253.514119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1253.538871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1253.576517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.616475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1253.646989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1253.676294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1253.703832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.738523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1254.229717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_error -r 1 [ 1254.255381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e remote_error -r 1 [ 1254.292054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_error -r 1 [ 1254.330544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e remote_error -r 1 [ 1254.380728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e remote_error -r 1 [ 1254.414961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e remote_error -r 1 [ 1254.455194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e remote_error -r 1 [ 1254.492627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e remote_error -r 1 [ 1254.531248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1254.550132] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1254.557053] LNet: Skipped 2 previous similar messages [ 1254.744879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.768603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1254.792270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.825753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.866115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.896356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.935450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.980801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.996616] LNet: 62213:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.009313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.045168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.070816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.101491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.123623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.141298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.164920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1255.189408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.621758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_timeout -r 1 [ 1255.651372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e remote_timeout -r 1 [ 1255.680251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_timeout -r 1 [ 1255.713458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e remote_timeout -r 1 [ 1255.743801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e remote_timeout -r 1 [ 1255.770429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e remote_timeout -r 1 [ 1255.801549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e remote_timeout -r 1 [ 1255.839069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e remote_timeout -r 1 [ 1255.868211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.121@tcp [ 1255.887895] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1255.892893] LNet: Skipped 1 previous similar message [ 1256.080252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.103588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1256.123676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.150772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.175907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.206955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.232346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.263656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.275307] LNet: 62336:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.292604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.316396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.343809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.376902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.405741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.431886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.457959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.489588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.917539] LNet: 62414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.924202] LNetError: 62414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.934071] LNet: Removed LNI 192.168.201.21@tcp [ 1260.945330] LNet: Removed LNI 192.168.201.21@tcp1 [ 1261.239342] Key type .llcrypt unregistered [ 1261.240467] Key type ._llcrypt unregistered [ 1273.842777] Key type ._llcrypt registered [ 1273.844692] Key type .llcrypt registered [ 1274.659623] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:48:53 (1782481733) [ 1275.728607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1276.946912] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:48:55 (1782481735) [ 1277.532328] Key type .llcrypt unregistered [ 1277.533768] Key type ._llcrypt unregistered [ 1282.913909] Key type ._llcrypt registered [ 1282.915642] Key type .llcrypt registered [ 1282.950140] LNet: 63742:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1282.950140] mask usage: [+|-] ... [ 1283.079626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.090324] alg: No test for adler32 (adler32-zlib) [ 1283.973830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.991946] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1290.530769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1291.046574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e network_timeout -r 1 [ 1291.067320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e network_timeout -r 1 [ 1291.086859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1291.099822] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1291.167103] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1291.325707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.356554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1291.383695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.413539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.440142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.455423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.481376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.507822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.517157] LNet: 63956:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.526990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.542994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.564265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.590640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.610737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.634727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.951771] LNet: 63995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.956800] LNetError: 63995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.030638] LNet: Removed LNI 192.168.201.21@tcp [ 1293.402242] Key type .llcrypt unregistered [ 1293.403638] Key type ._llcrypt unregistered [ 1306.729917] Key type ._llcrypt registered [ 1306.732418] Key type .llcrypt registered [ 1306.761820] LNet: 64444:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1306.761820] mask usage: [+|-] ... [ 1306.901058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1306.920769] alg: No test for adler32 (adler32-zlib) [ 1307.781804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1307.793895] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1313.791694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1321.110870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.159829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1321.185861] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1322.054481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e network_timeout -r 1 [ 1322.088412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e network_timeout -r 1 [ 1322.130651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e network_timeout -r 1 [ 1322.166348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e network_timeout -r 1 [ 1322.199709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e network_timeout -r 1 [ 1322.227875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e network_timeout -r 1 [ 1322.261609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e network_timeout -r 1 [ 1322.293702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e network_timeout -r 1 [ 1322.319630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1322.341762] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1322.540509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1322.565644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1322.586170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1322.613337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1322.644502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1322.663842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1322.686208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1322.708131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1322.720936] LNet: 64873:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1322.734525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1322.769752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1322.811138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.836824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1322.870297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1322.901612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1326.299039] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1326.512581] LNet: 64946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.521612] LNetError: 64946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.530279] LNet: Removed LNI 192.168.201.21@tcp [ 1326.544804] LNet: Removed LNI 192.168.201.21@tcp1 [ 1326.825499] Key type .llcrypt unregistered [ 1326.826817] Key type ._llcrypt unregistered [ 1337.450671] Key type ._llcrypt registered [ 1337.454093] Key type .llcrypt registered [ 1338.576439] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:49:56 (1782481796) [ 1338.749949] LNet: 65745:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1338.749949] mask usage: [+|-] ... [ 1338.864845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1338.881570] alg: No test for adler32 (adler32-zlib) [ 1339.719284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.754409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.780371] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1340.319213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.341969] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1340.976177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1341.055778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1341.098994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1341.128179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1341.159636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1341.213529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1341.235636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.21@tcp [ 1341.260239] LNetError: 66044:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.21@tcp to recovery [ 1341.791280] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1341.793516] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1343.839291] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1344.629240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1347.936839] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1349.144781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1349.200946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.302685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.337587] LNet: 66161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.346865] LNetError: 66161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.355718] LNet: Removed LNI 192.168.201.21@tcp [ 1349.372390] LNet: Removed LNI 192.168.201.21@tcp1 [ 1349.385746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.437765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.471269] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1350.015648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.032864] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1350.725183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1350.747955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.801661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1350.832707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1350.856938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1350.881656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1350.909403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1350.956278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.21@tcp [ 1350.972866] LNetError: 66434:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.21@tcp to recovery [ 1351.458441] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1354.409514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.599766] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1357.605150] LNet: Skipped 1 previous similar message [ 1362.370988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1362.410971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1362.436157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1368.114277] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:50:26 (1782481826) [ 1368.370569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.387588] LNet: 67150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.397566] LNetError: 67150:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.404157] LNetError: 67150:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1368.411924] LNet: Removed LNI 192.168.201.21@tcp [ 1368.426971] LNet: Removed LNI 192.168.201.21@tcp1 [ 1368.445488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.493645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.540581] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1369.090292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.108602] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1369.755707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1369.805526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1369.824607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1369.845455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1369.866764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1369.888180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1369.913425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.21@tcp --health 0 [ 1369.974269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1370.529026] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.21@tcp [ 1370.532420] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1370.540340] LNet: Skipped 1 previous similar message [ 1385.941460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1386.046472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1386.083990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.110523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.21@tcp --health 500 [ 1386.186461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.750646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1392.812699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.831931] LNet: 67650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.837407] LNetError: 67650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.842370] LNetError: 67650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1392.850066] LNet: Removed LNI 192.168.201.21@tcp [ 1392.870608] LNet: Removed LNI 192.168.201.21@tcp1 [ 1392.898308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.975138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.996079] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1393.467641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.483288] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1394.085934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1394.108495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1394.161089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1394.192483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1394.222268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1394.251697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1394.275730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1394.299821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.21@tcp --health 0 [ 1395.999781] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1396.004965] LNet: Skipped 3 previous similar messages [ 1396.628519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1405.664938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1405.710929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1405.739980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.652853] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:51:09 (1782481869) [ 1410.976224] LNet: 68695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.984570] LNetError: 68695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.989674] LNetError: 68695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1410.994277] LNet: Removed LNI 192.168.201.21@tcp [ 1411.004353] LNet: Removed LNI 192.168.201.21@tcp1 [ 1411.518541] Key type .llcrypt unregistered [ 1411.521414] Key type ._llcrypt unregistered [ 1416.358564] Key type ._llcrypt registered [ 1416.367503] Key type .llcrypt registered [ 1416.403482] LNet: 68918:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1416.403482] mask usage: [+|-] ... [ 1416.599806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1416.621152] alg: No test for adler32 (adler32-zlib) [ 1417.574607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1417.607681] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1424.918550] Lustre: DEBUG MARKER: Initial discovery [ 1424.981596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.121@tcp [ 1426.711935] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1426.938979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1426.994749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.121@tcp [ 1427.017871] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1427.046419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1427.092632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.121@tcp -r 1 -m GET -l 3 [ 1427.117147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.121@tcp -r 1 -m GET -e local_error [ 1427.141543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.121@tcp -r 1 -m PUT -l 6 [ 1427.874662] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1428.542764] Lustre: DEBUG MARKER: Force oleg121-client.virtnet to discover 192.168.201.121@tcp (in background) [ 1428.594275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.121@tcp [ 1429.837946] Lustre: DEBUG MARKER: Force oleg121-server to discover 192.168.201.21@tcp [ 1431.647341] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1431.831528] Lustre: DEBUG MARKER: Wait for 69448 [ 1433.027373] Lustre: DEBUG MARKER: Finished wait on 69448 [ 1433.332495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1433.367372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1433.368931] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1433.755760] LNet: 69906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.760652] LNetError: 69906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.777912] LNet: Removed LNI 192.168.201.21@tcp [ 1434.386417] Key type .llcrypt unregistered [ 1434.389270] Key type ._llcrypt unregistered [ 1449.144633] Key type ._llcrypt registered [ 1449.145956] Key type .llcrypt registered [ 1450.141179] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:51:48 (1782481908) [ 1450.647571] Key type .llcrypt unregistered [ 1450.650943] Key type ._llcrypt unregistered [ 1456.429520] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1456.593173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1456.597213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1456.605833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1459.043141] Key type ._llcrypt registered [ 1459.044794] Key type .llcrypt registered [ 1459.065601] LNet: 70949:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1459.065601] mask usage: [+|-] ... [ 1459.167977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.180844] alg: No test for adler32 (adler32-zlib) [ 1460.051317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.098964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.121365] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1460.895432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1460.910994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1468.372689] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:52:06 (1782481926) [ 1468.543661] LNet: 71974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.558213] LNetError: 71974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1468.565548] LNet: Removed LNI 10.1.2.3@tcp [ 1468.583460] LNet: Removed LNI 192.168.201.21@tcp [ 1469.390432] Key type .llcrypt unregistered [ 1469.392437] Key type ._llcrypt unregistered [ 1475.847192] Key type ._llcrypt registered [ 1475.853211] Key type .llcrypt registered [ 1475.883529] LNet: 72234:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1475.883529] mask usage: [+|-] ... [ 1476.005907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1476.017462] alg: No test for adler32 (adler32-zlib) [ 1476.940485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.993324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.008243] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1477.536936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1477.562501] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1478.380286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.21@tcp [ 1479.455460] LNetError: Unexpected error -22 connecting to 192.168.201.21@tcp at host 192.168.201.21:988 [ 1483.794530] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:52:22 (1782481942) [ 1483.858782] LNet: 73107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.862473] LNetError: 73107:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1484.896345] LNet: Removed LNI 10.1.2.3@tcp [ 1484.899562] LNetError: 73107:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.913255] LNet: Removed LNI 192.168.201.21@tcp [ 1485.354520] Key type .llcrypt unregistered [ 1485.355760] Key type ._llcrypt unregistered [ 1490.603970] Key type ._llcrypt registered [ 1490.606991] Key type .llcrypt registered [ 1490.662613] LNet: 73348:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1490.662613] mask usage: [+|-] ... [ 1490.823028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.848255] alg: No test for adler32 (adler32-zlib) [ 1491.735918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.771704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.788068] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1492.302444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1492.318223] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1492.973735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.21@tcp1 --nid 192.168.201.21@tcp2 [ 1496.060418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp1 192.168.201.21@tcp2 [ 1496.080395] LNetError: 73952:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.21@tcp1 to 192.168.201.21@tcp2 - Route table may be misconfigured [ 1496.107279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp2 192.168.201.21@tcp1 [ 1502.299428] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:52:40 (1782481960) [ 1502.428353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.461368] LNet: 74523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.472228] LNetError: 74523:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1502.480328] LNet: Removed LNI 192.168.201.21@tcp1 [ 1503.530047] LNet: Removed LNI 192.168.201.21@tcp2 [ 1503.590869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.685222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1503.705780] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1504.500319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1504.526311] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1505.252127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1505.291996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.21@tcp [ 1505.326417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp1 192.168.201.21@tcp1 [ 1505.372520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.21@tcp -d 192.168.201.21@tcp -e network_timeout [ 1505.410694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -e network_timeout [ 1505.442982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -e network_timeout [ 1505.475202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -e network_timeout [ 1505.508253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1505.526903] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1505.547195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1505.637355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1505.696092] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1505.708054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1510.744766] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:52:49 (1782481969) [ 1510.897090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.918054] LNet: 75387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.928259] LNetError: 75387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.933062] LNetError: 75387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1510.958244] LNet: Removed LNI 192.168.201.21@tcp [ 1511.984897] LNet: Removed LNI 192.168.201.21@tcp1 [ 1512.003446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.143913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1512.315501] LNet: 75425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.618287] Key type .llcrypt unregistered [ 1512.620095] Key type ._llcrypt unregistered [ 1520.688062] Key type ._llcrypt registered [ 1520.689431] Key type .llcrypt registered [ 1521.759942] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:53:00 (1782481980) [ 1521.995819] LNet: 76195:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1521.995819] mask usage: [+|-] ... [ 1522.137762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1522.156653] alg: No test for adler32 (adler32-zlib) [ 1523.056176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.390464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1525.515838] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:53:03 (1782481983) [ 1525.704263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.723495] LNet: 76522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.741853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.795891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.819613] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1526.318755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1526.331361] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1526.941118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1526.965529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp1 [ 1527.001603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp1 [ 1531.088567] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:53:09 (1782481989) [ 1532.720810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg121-server [ 1533.854799] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:53:12 (1782481992) [ 1535.305281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg121-server [ 1536.491523] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:53:14 (1782481994) [ 1537.828186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg121-server [ 1539.048810] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:53:17 (1782481997) [ 1540.410587] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg121-server [ 1541.932245] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:53:20 (1782482000) [ 1543.263948] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg121-server [ 1544.848967] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:53:22 (1782482002) [ 1546.453615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg121-server [ 1548.097081] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:53:26 (1782482006) [ 1549.783600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg121-server [ 1551.110086] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:53:29 (1782482009) [ 1552.661259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg121-server [ 1554.331484] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:53:32 (1782482012) [ 1555.786668] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg121-server [ 1557.208723] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:53:35 (1782482015) [ 1557.422876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.446800] LNet: 80398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.452930] LNetError: 80398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.457033] LNet: Removed LNI 192.168.201.21@tcp [ 1558.506289] LNet: Removed LNI 192.168.201.21@tcp1 [ 1558.537624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.595724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.613832] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1559.222451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1559.318584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1560.644595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.665713] LNet: 80554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.674770] LNetError: 80554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.680349] LNetError: 80554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1561.704504] LNet: Removed LNI 192.168.201.21@tcp [ 1561.727352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.776509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1561.800849] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1562.408549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1562.558538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.582774] LNetError: 80699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.593884] LNet: Removed LNI 192.168.201.21@tcp [ 1562.623689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.676437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.703611] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1563.269765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1563.326788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1563.439542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.450128] LNet: 80847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.454389] LNet: 80847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1564.522473] LNet: Removed LNI 192.168.201.21@tcp [ 1564.543886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.572905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1564.588830] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1565.068368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1565.211453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.227421] LNetError: 80992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1565.232076] LNetError: 80992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1566.246152] LNet: Removed LNI 192.168.201.21@tcp [ 1566.284474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.357288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.382120] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1566.949496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1567.007822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1567.169616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.242978] LNet: Removed LNI 192.168.201.21@tcp [ 1568.274622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.322955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1568.340526] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1568.923920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1569.086657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.104356] LNet: 81284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.107369] LNet: 81284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1570.150928] LNet: Removed LNI 192.168.201.21@tcp [ 1570.172546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.235187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1570.255025] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1570.809913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1570.886848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1571.070124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.095507] LNetError: 81432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.103029] LNetError: 81432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1572.133718] LNet: Removed LNI 192.168.201.21@tcp [ 1572.148784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.250528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1572.279791] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1572.882401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1573.060751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.124299] LNet: Removed LNI 192.168.201.21@tcp [ 1574.157559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.229326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.267198] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1574.851797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1574.921580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1575.123157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.166950] LNet: Removed LNI 192.168.201.21@tcp [ 1576.189167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.249595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1576.270443] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1576.818866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1577.028712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.087770] LNet: Removed LNI 192.168.201.21@tcp [ 1578.122637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.158982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.174588] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1578.672172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1578.732645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1578.899492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.924695] LNet: 82017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.929770] LNet: 82017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1580.008038] LNet: Removed LNI 192.168.201.21@tcp [ 1580.033815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.081464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1580.097582] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1580.595738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1580.740777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.755267] LNetError: 82163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.758080] LNetError: 82163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1580.766922] LNet: Removed LNI 192.168.201.21@tcp [ 1580.779917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.817817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.838320] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1581.343409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1581.370323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.385845] LNet: Removed LNI 192.168.201.21@tcp [ 1581.398352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.433715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.449696] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1581.970819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1587.260930] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:54:05 (1782482045) [ 1587.344769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.372761] LNet: Removed LNI 192.168.201.21@tcp [ 1587.412620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.470148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1587.486767] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/180] [ 1588.341384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.378280] LNet: Removed LNI 192.168.201.21@tcp231 [ 1588.396398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.437329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1588.454074] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/180] [ 1588.459900] LNet: 83145:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1588.755502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1588.780341] LNet: Removed LNI 192.168.201.21@tcp231 [ 1588.799712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1588.822073] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/180] [ 1589.584369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.625403] LNet: Removed LNI 192.168.201.21@tcp231 [ 1589.651538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.708762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1589.724845] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/0] [ 1589.729744] LNet: 83297:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.945891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1589.966861] LNet: Removed LNI 192.168.201.21@tcp231 [ 1589.974664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1589.995908] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/0] [ 1590.760679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.781785] LNet: Removed LNI 192.168.201.21@tcp231 [ 1590.802500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.852430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1590.875895] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/60] [ 1590.881027] LNet: 83449:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.204420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1591.232441] LNet: Removed LNI 192.168.201.21@tcp231 [ 1591.249824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1591.273577] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/60] [ 1592.168385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.195994] LNet: Removed LNI 192.168.201.21@tcp231 [ 1592.215335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.257237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1592.281608] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/180] [ 1592.593708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1592.623331] LNet: Removed LNI 192.168.201.21@tcp231 [ 1592.635789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1592.657089] LNet: Added LNI 192.168.201.21@tcp231 [8/256/0/180] [ 1598.715458] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:54:16 (1782482056) [ 1598.969485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.998992] LNet: 84310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.005199] LNet: 84310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 1599.011551] LNetError: 84310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.018153] LNetError: 84310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1599.032907] LNet: Removed LNI 192.168.201.21@tcp231 [ 1599.051615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.112105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1599.134884] LNet: Added LNI 192.168.201.21@tcp232 [8/256/0/180] [ 1599.782698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1599.826202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1599.866388] LNet: Removed LNI 192.168.201.21@tcp232 [ 1599.896551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1599.916350] LNet: Added LNI 192.168.201.21@tcp232 [8/256/0/180] [ 1600.520356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1600.556934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1600.590856] LNet: Removed LNI 192.168.201.21@tcp232 [ 1600.619748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1600.649566] LNet: Added LNI 192.168.201.21@tcp232 [8/256/0/180] [ 1601.244344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1601.298648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1601.327270] LNet: Removed LNI 192.168.201.21@tcp232 [ 1601.346620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1601.367994] LNet: Added LNI 192.168.201.21@tcp232 [8/256/0/180] [ 1602.078683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1602.110414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1602.136793] LNet: Removed LNI 192.168.201.21@tcp232 [ 1602.156426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1606.986892] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:54:25 (1782482065) [ 1607.997483] Key type .llcrypt unregistered [ 1608.000296] Key type ._llcrypt unregistered [ 1612.087508] Key type ._llcrypt registered [ 1612.089088] Key type .llcrypt registered [ 1612.111376] LNet: 85667:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1612.111376] mask usage: [+|-] ... [ 1612.257778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1612.268441] alg: No test for adler32 (adler32-zlib) [ 1613.110414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1613.123988] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1618.376628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1624.456675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.499631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.516590] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1625.520623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_interrupt -r 2 [ 1625.538807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_interrupt -r 2 [ 1625.562609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_interrupt -r 2 [ 1625.585785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_interrupt -r 2 [ 1625.614408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_interrupt -r 2 [ 1625.644242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_interrupt -r 2 [ 1625.668872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_interrupt -r 2 [ 1625.693826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_interrupt -r 2 [ 1625.725860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1625.743819] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1625.749552] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1625.911866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.938746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1625.964817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.993958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1626.027976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1626.053514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1626.080729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1626.107736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1626.120580] LNet: 86099:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1626.134279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1626.157523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1626.179981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.210851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.259260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.285307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.713836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_dropped -r 2 [ 1626.741981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_dropped -r 2 [ 1626.767356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_dropped -r 2 [ 1626.801208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_dropped -r 2 [ 1626.843670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_dropped -r 2 [ 1626.868366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_dropped -r 2 [ 1626.891617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_dropped -r 2 [ 1626.913354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_dropped -r 2 [ 1626.940394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1626.958408] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1627.123356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1627.148404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1627.177912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1627.198945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.219669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.246798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.268819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.294839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.309497] LNet: 86219:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.328245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.355660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1627.375797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.404937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1627.426714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1627.446455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1627.847428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_aborted -r 2 [ 1627.882246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_aborted -r 2 [ 1627.912886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_aborted -r 2 [ 1627.944860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_aborted -r 2 [ 1627.970898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_aborted -r 2 [ 1628.004389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_aborted -r 2 [ 1628.045469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_aborted -r 2 [ 1628.081355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_aborted -r 2 [ 1628.122932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1628.142659] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1628.309254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1628.327881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1628.351200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1628.368684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1628.386637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1628.405789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1628.431498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1628.460942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1628.477065] LNet: 86339:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1628.491749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1628.513214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1628.540816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1628.566992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1628.588524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1628.611540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1628.966281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_no_route -r 2 [ 1628.995353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_no_route -r 2 [ 1629.022717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_no_route -r 2 [ 1629.048410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_no_route -r 2 [ 1629.076554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_no_route -r 2 [ 1629.119205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_no_route -r 2 [ 1629.161522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_no_route -r 2 [ 1629.189660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_no_route -r 2 [ 1629.224222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1629.432791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1629.455684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1629.480411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1629.513129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1629.542938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1629.570273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1629.600987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1629.634257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1629.656772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1629.679960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1629.700076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1629.726454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1629.748383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1629.768924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1630.186432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e local_timeout -r 2 [ 1630.214468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e local_timeout -r 2 [ 1630.253926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e local_timeout -r 2 [ 1630.284756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e local_timeout -r 2 [ 1630.316922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e local_timeout -r 2 [ 1630.342993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e local_timeout -r 2 [ 1630.379517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e local_timeout -r 2 [ 1630.408634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e local_timeout -r 2 [ 1630.437192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1630.451684] LNet: There was an unexpected network error while writing to 192.168.201.121: rc = -22 [ 1630.455042] LNet: Skipped 1 previous similar message [ 1630.623119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1630.647687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1630.687601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1630.711846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1630.742227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1630.769548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1630.790487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1630.817840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1630.840446] LNet: 86581:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1630.843972] LNet: 86581:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1630.867581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1630.890709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1630.913071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1630.937983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.964758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1630.994543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1631.482535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp -m GET -e remote_dropped -r 2 [ 1631.523784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.121@tcp1 -m GET -e remote_dropped -r 2 [ 1631.544267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp -m GET -e remote_dropped -r 2 [ 1631.570283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp -d 192.168.201.21@tcp1 -m GET -e remote_dropped -r 2 [ 1631.596360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp -m GET -e remote_dropped -r 2 [ 1631.629580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.121@tcp1 -m GET -e remote_dropped -r 2 [ 1631.664578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp -m GET -e remote_dropped -r 2 [ 1631.690841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.21@tcp1 -d 192.168.201.21@tcp1 -m GET -e remote_dropped -r 2 [ 1631.715695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.21@tcp 192.168.201.121@tcp [ 1631.734440] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.121@tcp [ 1631.954909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1631.979906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1632.001404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1632.021459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1632.043543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1632.066856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1632.096476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1632.140932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1632.168888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1632.194941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.218107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.240382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.263828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.283346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.030401] LNet: 86777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.034576] LNetError: 86777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.039881] LNet: Removed LNI 192.168.201.21@tcp [ 1636.051385] LNet: Removed LNI 192.168.201.21@tcp1 [ 1636.317712] Key type .llcrypt unregistered [ 1636.320890] Key type ._llcrypt unregistered [ 1648.326792] Key type ._llcrypt registered [ 1648.328713] Key type .llcrypt registered [ 1649.207409] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:55:07 (1782482107) [ 1649.398908] LNet: 87582:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1649.398908] mask usage: [+|-] ... [ 1649.521362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1649.536953] alg: No test for adler32 (adler32-zlib) [ 1650.407095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.455744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.474539] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1651.161557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1651.186717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.21@tcp [ 1651.204141] LNetError: 87754:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.21@tcp to recovery [ 1651.271156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.21@tcp [ 1651.347411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1655.951928] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:55:14 (1782482114) [ 1656.052325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.076881] LNet: 88337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.080820] LNetError: 88337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.090265] LNet: Removed LNI 192.168.201.21@tcp [ 1656.106771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.177936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.220694] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1656.701993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.716607] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1657.488602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.21@tcp [ 1657.505772] LNetError: 88609:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.201.21@tcp to recovery [ 1658.207304] LNet: 1 local NIs in recovery (showing 1): 192.168.201.21@tcp [ 1658.221384] LNet: There was an unexpected network error while writing to 192.168.201.21: rc = -22 [ 1658.692066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.21@tcp [ 1665.400671] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:55:23 (1782482123) [ 1666.785258] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg121-server [ 1668.123214] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:55:26 (1782482126) [ 1669.541731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg121-server [ 1670.928469] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:55:29 (1782482129) [ 1672.552285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg121-server [ 1674.012815] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:55:32 (1782482132) [ 1674.190318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.229295] LNet: 90239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.238215] LNetError: 90239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.248612] LNet: Removed LNI 192.168.201.21@tcp [ 1674.268618] LNet: Removed LNI 192.168.201.21@tcp1 [ 1674.290759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.352333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.374795] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1675.080895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1675.152709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1675.204417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1675.267750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1675.291708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1675.327928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1679.708590] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:55:38 (1782482138) [ 1679.907387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.924062] LNet: 90964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.928823] LNetError: 90964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.933805] LNetError: 90964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1679.947451] LNet: Removed LNI 192.168.201.21@tcp [ 1679.961528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.012661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.028914] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1685.175902] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:55:43 (1782482143) [ 1685.339116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.360444] LNet: 91667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.365772] LNetError: 91667:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.376140] LNet: Removed LNI 192.168.201.21@tcp [ 1685.394383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.433349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.445713] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1690.044488] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:55:48 (1782482148) [ 1691.188039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1692.425634] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:55:50 (1782482150) [ 1692.897324] LNet: 92701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.900403] LNetError: 92701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1692.909982] LNet: Removed LNI 192.168.201.21@tcp [ 1693.301961] Key type .llcrypt unregistered [ 1693.303645] Key type ._llcrypt unregistered [ 1697.568888] Key type ._llcrypt registered [ 1697.571623] Key type .llcrypt registered [ 1697.597793] LNet: 92930:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1697.597793] mask usage: [+|-] ... [ 1697.715597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1697.732931] alg: No test for adler32 (adler32-zlib) [ 1698.636807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1698.665384] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1704.496967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1707.573715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.121@tcp [ 1707.807734] LNet: 93117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1707.813855] LNetError: 93117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.825754] LNet: Removed LNI 192.168.201.21@tcp [ 1708.132358] Key type .llcrypt unregistered [ 1708.136143] Key type ._llcrypt unregistered [ 1716.093291] Key type ._llcrypt registered [ 1716.095017] Key type .llcrypt registered [ 1717.035609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1718.180348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1719.351615] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:56:17 (1782482177) [ 1720.390385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1721.541416] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:56:20 (1782482180) [ 1722.929823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg121-server [ 1724.107947] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:56:22 (1782482182) [ 1725.495447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg121-server [ 1726.647899] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:56:25 (1782482185) [ 1727.463252] LNet: 95167:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1727.463252] mask usage: [+|-] ... [ 1727.917600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.956661] alg: No test for adler32 (adler32-zlib) [ 1728.969780] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 1729.149577] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1730.815323] Key type lgssc registered [ 1731.771102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1746.093950] Key type lgssc unregistered [ 1746.389971] LNet: 96289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.398847] LNetError: 96289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.421501] LNet: Removed LNI 192.168.201.21@tcp [ 1746.910970] Key type .llcrypt unregistered [ 1746.914854] Key type ._llcrypt unregistered [ 1758.150428] Key type ._llcrypt registered [ 1758.152493] Key type .llcrypt registered [ 1759.244724] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:56:57 (1782482217) [ 1759.415803] LNet: 97085:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1759.415803] mask usage: [+|-] ... [ 1759.524623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.548690] alg: No test for adler32 (adler32-zlib) [ 1760.389657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.425635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1760.456949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1760.487484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1760.521444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1765.209926] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:57:03 (1782482223) [ 1765.437458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.457589] LNet: 97702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.474553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.518292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.539202] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1766.368393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.386446] LNet: 97846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.399301] LNetError: 97846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.415168] LNet: Removed LNI 192.168.201.21@tcp [ 1766.442483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.490523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1766.509398] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1766.827946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.849656] LNet: Removed LNI 192.168.201.21@tcp [ 1766.865751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.919246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1766.944477] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1772.030338] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:57:10 (1782482230) [ 1772.133730] LNet: 98460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.141952] LNet: 98460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1772.147516] LNetError: 98460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.152030] LNetError: 98460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1772.167422] LNet: Removed LNI 192.168.201.21@tcp [ 1772.645440] Key type .llcrypt unregistered [ 1772.646915] Key type ._llcrypt unregistered [ 1777.433502] Key type ._llcrypt registered [ 1777.435952] Key type .llcrypt registered [ 1777.479913] LNet: 98695:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1777.479913] mask usage: [+|-] ... [ 1777.648134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1777.681682] alg: No test for adler32 (adler32-zlib) [ 1778.564669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.625817] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1778.628737] LNet: *** cfs_fail_loc=e003, val=100*** [ 1779.256224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1779.277230] LNet: *** cfs_fail_loc=e003, val=100*** [ 1779.279081] LNet: Skipped 2 previous similar messages [ 1779.288204] Ping reply received with mapping: [ 1779.291217] found metadata reply NI[192.168.201.21@tcp] [ 1779.293475] 15 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1779.296849] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1779.298964] 5a 5a 5a 5a 5a 5a 5a 5a [ 1779.386264] LNet: 98862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.389513] LNetError: 98862:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.397469] LNet: Removed LNI 192.168.201.21@tcp [ 1779.783712] Key type .llcrypt unregistered [ 1779.786228] Key type ._llcrypt unregistered [ 1788.283795] Key type ._llcrypt registered [ 1788.286192] Key type .llcrypt registered [ 1789.281879] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:57:27 (1782482247) [ 1789.894281] Key type .llcrypt unregistered [ 1789.899502] Key type ._llcrypt unregistered [ 1795.829736] Key type ._llcrypt registered [ 1795.832794] Key type .llcrypt registered [ 1795.874989] LNet: 99857:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1795.874989] mask usage: [+|-] ... [ 1796.090654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1796.120602] alg: No test for adler32 (adler32-zlib) [ 1797.216963] LNetError: 99907:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1797.622468] Key type .llcrypt unregistered [ 1797.624578] Key type ._llcrypt unregistered [ 1802.314651] Key type ._llcrypt registered [ 1802.316462] Key type .llcrypt registered [ 1803.491669] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:57:41 (1782482261) [ 1803.954301] Key type .llcrypt unregistered [ 1803.955924] Key type ._llcrypt unregistered [ 1808.471217] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1808.477277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1810.929130] Key type ._llcrypt registered [ 1810.931799] Key type .llcrypt registered [ 1810.963383] LNet: 100775:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1810.963383] mask usage: [+|-] ... [ 1811.115512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1811.130174] alg: No test for adler32 (adler32-zlib) [ 1812.074960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.120498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1819.604804] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:57:58 (1782482278) [ 1819.751411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.765321] LNet: 101687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.772361] LNetError: 101687:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1819.784642] LNet: Removed LNI 10.1.2.3@tcp [ 1819.798712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.299573] Key type .llcrypt unregistered [ 1820.301507] Key type ._llcrypt unregistered [ 1824.804940] Key type ._llcrypt registered [ 1824.806479] Key type .llcrypt registered [ 1824.833500] LNet: 101918:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1824.833500] mask usage: [+|-] ... [ 1824.935278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1824.948582] alg: No test for adler32 (adler32-zlib) [ 1825.827523] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1825.877042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1826.021141] LNet: 101976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.026361] LNetError: 101976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.043464] LNet: Removed LNI 192.168.201.21@tcp [ 1826.551652] Key type .llcrypt unregistered [ 1826.555254] Key type ._llcrypt unregistered [ 1835.626824] Key type ._llcrypt registered [ 1835.628696] Key type .llcrypt registered [ 1836.760782] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:58:14 (1782482294) [ 1837.174932] Key type .llcrypt unregistered [ 1837.178953] Key type ._llcrypt unregistered [ 1841.519698] Key type ._llcrypt registered [ 1841.522117] Key type .llcrypt registered [ 1841.557845] LNet: 102959:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1841.557845] mask usage: [+|-] ... [ 1841.685500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1841.699210] alg: No test for adler32 (adler32-zlib) [ 1847.256660] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:58:25 (1782482305) [ 1847.346825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.393764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.453993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.492738] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1848.019637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.046962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1848.070354] LNet: Removed LNI 192.168.201.21@tcp [ 1848.276568] LNet: 103718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.615696] Key type .llcrypt unregistered [ 1848.617239] Key type ._llcrypt unregistered [ 1857.146947] Key type ._llcrypt registered [ 1857.149817] Key type .llcrypt registered [ 1858.242125] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:58:36 (1782482316) [ 1858.516875] LNet: 104482:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1858.516875] mask usage: [+|-] ... [ 1858.609196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1858.619304] alg: No test for adler32 (adler32-zlib) [ 1859.489240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.524476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1859.550082] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1860.080486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1860.112825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.21@tcp [ 1860.125860] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1860.129982] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1860.132620] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1860, 0, 0 [ 1860.136757] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1860.139435] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1860.142468] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1860.145247] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1860, 0, 0 [ 1860.148865] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1860.151573] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1860.154478] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1860.157178] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1860, 0, 1910 [ 1860.164541] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1860.166458] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1860.171870] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1860.174644] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1860, 0, 1910 [ 1860.177867] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1865.504431] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:58:43 (1782482323) [ 1865.836727] LNet: 105256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.844685] LNetError: 105256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.855189] LNet: Removed LNI 192.168.201.21@tcp [ 1866.322432] Key type .llcrypt unregistered [ 1866.323833] Key type ._llcrypt unregistered [ 1871.417601] Key type ._llcrypt registered [ 1871.420667] Key type .llcrypt registered [ 1871.466940] LNet: 105479:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1871.466940] mask usage: [+|-] ... [ 1871.691283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1871.730753] alg: No test for adler32 (adler32-zlib) [ 1872.658683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1872.682263] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1878.456593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 1885.075518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.111352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1885.136458] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 1885.864398] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1885.958443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1885.964544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1885.969107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1888.280331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1888.305448] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1895.960465] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:59:14 (1782482354) [ 1896.025537] LNet: 106747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.033492] LNetError: 106747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.041252] LNet: Removed LNI 192.168.201.21@tcp [ 1897.056511] LNet: Removed LNI 192.168.201.21@tcp1 [ 1897.059643] LNetError: 106747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1897.064456] LNetError: 106747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1897.084815] LNet: Removed LNI 10.1.2.3@tcp99 [ 1897.554444] Key type .llcrypt unregistered [ 1897.556211] Key type ._llcrypt unregistered [ 1902.160374] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1902.165137] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1904.690888] Key type ._llcrypt registered [ 1904.693282] Key type .llcrypt registered [ 1904.721410] LNet: 107021:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1904.721410] mask usage: [+|-] ... [ 1904.858606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1904.867380] alg: No test for adler32 (adler32-zlib) [ 1905.725302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.765227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.784753] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1906.313334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1906.328242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1907.844220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1907.882966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1907.922337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1907.936038] LNetError: 107500:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1907.955477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1907.993304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1908.269996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1908.337414] LNet: 107520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.341982] LNetError: 107520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1908.347825] LNet: Removed LNI 10.1.2.3@tcp [ 1909.413864] LNet: Removed LNI 192.168.201.21@tcp [ 1909.859421] Key type .llcrypt unregistered [ 1909.868364] Key type ._llcrypt unregistered [ 1918.350461] Key type ._llcrypt registered [ 1918.353166] Key type .llcrypt registered [ 1919.744323] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:59:37 (1782482377) [ 1919.874793] LNet: 108314:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1919.874793] mask usage: [+|-] ... [ 1920.012554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1920.029649] alg: No test for adler32 (adler32-zlib) [ 1920.923699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.976505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.012832] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1921.744724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.21@tcp [ 1921.815194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg121-client@tcp [ 1926.533892] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:59:44 (1782482384) [ 1926.734667] LNet: 109053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1926.741446] LNetError: 109053:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1926.752147] LNet: Removed LNI 192.168.201.21@tcp [ 1927.301417] Key type .llcrypt unregistered [ 1927.303724] Key type ._llcrypt unregistered [ 1931.502894] Key type ._llcrypt registered [ 1931.504887] Key type .llcrypt registered [ 1931.641594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.652538] alg: No test for adler32 (adler32-zlib) [ 1932.535380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1932.551603] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1932.574452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1936.837132] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:59:55 (1782482395) [ 1938.059163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1939.328808] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:59:57 (1782482397) [ 1940.521091] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1942.023698] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:00:00 (1782482400) [ 1942.171541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.195748] LNet: 110469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.204953] LNetError: 110469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.213970] LNet: Removed LNI 192.168.201.21@tcp [ 1942.226708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.269489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1942.287787] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1942.782469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1942.814686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1947.308856] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:00:05 (1782482405) [ 1947.427406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.470859] LNet: 111172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.488026] LNetError: 111172:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.512150] LNet: Removed LNI 192.168.201.21@tcp [ 1947.572416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.626846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1947.650361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1948.107468] Key type .llcrypt unregistered [ 1948.109096] Key type ._llcrypt unregistered [ 1955.923198] Key type ._llcrypt registered [ 1955.925163] Key type .llcrypt registered [ 1956.774104] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:00:15 (1782482415) [ 1956.984839] LNet: 111985:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1956.984839] mask usage: [+|-] ... [ 1957.115729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.137226] alg: No test for adler32 (adler32-zlib) [ 1958.032743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.067961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.098298] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1958.525814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1958.559981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1962.435569] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:00:20 (1782482420) [ 1962.564575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.588446] LNet: 112734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.597853] LNetError: 112734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1962.611611] LNet: Removed LNI 192.168.201.21@tcp [ 1962.623955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.651547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1962.674523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1966.926668] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:00:25 (1782482425) [ 1967.006592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.018954] LNet: 113310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.039648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.090821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1967.357324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1967.398774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1967.432976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1967.668544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1967.696440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1967.758984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1967.785877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.850161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1968.164464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.186081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1968.225755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1968.462425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1968.481112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1973.161749] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:00:31 (1782482431) [ 1973.264373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.286056] LNet: 113946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.291068] LNet: 113946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1973.314047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.367619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.382435] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1973.966533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1973.981800] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1974.625661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1974.665882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1974.817793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.835750] LNetError: 114226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.903224] LNet: Removed LNI 192.168.201.21@tcp [ 1975.905774] LNetError: 114226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.920906] LNet: Removed LNI 192.168.201.21@tcp2 [ 1975.956501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.991218] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1975.996520] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1976.000310] LNet: 114232:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1976.340804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1976.498508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.512739] LNet: 114269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.517776] LNet: 114269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1977.567631] LNet: Removed LNI 192.168.201.21@tcp [ 1977.578995] LNet: Removed LNI 192.168.201.21@tcp2 [ 1977.597765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.646335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.661842] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1978.165336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1978.179718] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1978.756687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1978.782165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1978.821692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1978.965601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.978926] LNetError: 114554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.983766] LNetError: 114554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1978.987803] LNet: Removed LNI 192.168.201.21@tcp [ 1980.006596] LNet: Removed LNI 192.168.201.21@tcp2 [ 1980.027739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.071899] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1980.077565] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1980.080991] LNet: 114560:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1980.611231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1980.902740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.920409] LNet: 114597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.924891] LNet: 114597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1980.929651] LNet: Removed LNI 192.168.201.21@tcp [ 1981.994242] LNet: Removed LNI 192.168.201.21@tcp2 [ 1982.012833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.052516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1982.079202] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1982.710660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1982.736064] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1983.413465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1983.477623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1983.621754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.634367] LNetError: 114876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.639215] LNetError: 114876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1983.643439] LNet: Removed LNI 192.168.201.21@tcp [ 1983.654362] LNet: Removed LNI 192.168.201.21@tcp2 [ 1983.680352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.729568] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1983.735718] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1983.739696] LNet: 114882:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1984.067551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1984.199553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.247303] LNet: Removed LNI 192.168.201.21@tcp [ 1985.259767] LNet: Removed LNI 192.168.201.21@tcp2 [ 1985.294341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.376511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.402667] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1986.047716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1986.072614] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1986.705967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1986.738223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1986.781467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1986.940552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.000096] LNet: Removed LNI 192.168.201.21@tcp [ 1989.030174] LNet: Removed LNI 192.168.201.21@tcp2 [ 1989.054830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.115591] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1989.122848] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1989.128339] LNet: 115210:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1989.706907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1989.882757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.902290] LNet: 115247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.906125] LNet: 115247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1989.915393] LNet: Removed LNI 192.168.201.21@tcp [ 1990.960343] LNet: Removed LNI 192.168.201.21@tcp2 [ 1990.990701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.093781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1991.117863] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1991.830969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1991.859472] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1992.475497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1992.496556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1992.538472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1992.795966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.826241] LNetError: 115544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.832599] LNetError: 115544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1992.839151] LNet: Removed LNI 192.168.201.21@tcp [ 1992.852221] LNet: Removed LNI 192.168.201.21@tcp2 [ 1992.869981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.923116] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1992.928278] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1993.266765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1993.553354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1994.591174] LNet: Removed LNI 192.168.201.21@tcp [ 1994.600942] LNet: Removed LNI 192.168.201.21@tcp2 [ 1994.630854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.683379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.714571] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1995.362305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1995.382069] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1995.944906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1995.972848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1995.990627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1996.033549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 1996.307501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1997.343942] LNet: Removed LNI 192.168.201.21@tcp [ 1998.375817] LNet: Removed LNI 192.168.201.21@tcp2 [ 1998.408393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.490326] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 1998.504610] LNet: Added LNI 192.168.201.21@tcp2 [8/256/0/180] [ 1998.511808] LNet: 115910:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1998.521643] LNet: 115910:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1999.005505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 2003.699368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2004.722513] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:01:03 (1782482463) [ 2004.950275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.978822] LNet: Removed LNI 192.168.201.21@tcp [ 2006.055167] LNet: Removed LNI 192.168.201.21@tcp2 [ 2006.085815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.309727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2006.327118] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2006.827725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2006.845404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.457474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.481746] LNet: 116954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.486750] LNet: 116954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2007.492609] LNet: Removed LNI 10.1.2.3@tcp [ 2007.509848] LNet: Removed LNI 192.168.201.21@tcp [ 2007.538586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.580865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2007.593617] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2007.597195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.810900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2007.827507] LNet: Removed LNI 192.168.201.21@tcp [ 2007.836341] LNet: Removed LNI 10.1.2.3@tcp [ 2012.234659] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:01:10 (1782482470) [ 2012.311899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.359765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.484189] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2012.586708] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2012.589411] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2012.593413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2014.915780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2014.940977] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2015.658159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2015.686155] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2017.177903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.21@tcp [ 2017.207102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2017.246541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.21@tcp -m PUT -e local_timeout -r 1 [ 2017.292920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2017.349077] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2017.355306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2017.550186] LNetError: 117620:0:(lib-move.c:4714:lnet_parse()) 192.168.201.21@tcp, src 192.168.201.21@tcp: Dropping PUT (error -108 looking up sender) [ 2018.592116] LNetError: 118079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2018.601576] LNetError: 118079:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2018.608904] LNet: Removed LNI 10.1.2.3@tcp [ 2019.627744] LNet: Removed LNI 192.168.201.21@tcp [ 2020.002679] Key type .llcrypt unregistered [ 2020.004600] Key type ._llcrypt unregistered [ 2027.960315] Key type ._llcrypt registered [ 2027.962450] Key type .llcrypt registered [ 2028.864269] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:01:27 (1782482487) [ 2029.013615] LNet: 118847:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2029.013615] mask usage: [+|-] ... [ 2029.106535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.119603] alg: No test for adler32 (adler32-zlib) [ 2029.951526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.212101] LNet: 118931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.623531] Key type .llcrypt unregistered [ 2030.630152] Key type ._llcrypt unregistered [ 2035.326230] Key type ._llcrypt registered [ 2035.327962] Key type .llcrypt registered [ 2035.359920] LNet: 119160:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2035.359920] mask usage: [+|-] ... [ 2035.512106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2035.528650] alg: No test for adler32 (adler32-zlib) [ 2036.396327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2036.410777] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2041.949314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 2042.099533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2042.508720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.121@tcp [ 2042.593419] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.121@tcp [ 2042.692903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2042.716608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2042.739955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2042.762316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2042.784043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2042.802514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2042.823568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2042.846550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2042.859858] LNet: 119380:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2042.875938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2042.899927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2042.926958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2042.946622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2042.964956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2042.988772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2043.368781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2043.488882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2043.517453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2043.552901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2043.584818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2043.611131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2043.642347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2043.670379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.695555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2043.707763] LNet: 119497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2043.726248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2043.748756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.778528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.799947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.822598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.851219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2044.387841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.121@tcp [ 2044.574318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2044.601733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2044.625244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2044.654699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2044.689265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2044.707314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2044.742414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2044.780405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2044.796289] LNet: 119614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2044.811532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2044.829912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2044.851546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2044.889814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2044.926970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2044.957490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2045.436309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2045.625322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2045.643544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2045.665876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2045.687511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2045.710935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2045.737141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2045.759493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2045.774731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2045.794308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2045.809890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2045.823335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2045.840964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2045.862299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2045.881580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2046.036873] LNet: 119777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.045312] LNetError: 119777:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.059651] LNet: Removed LNI 192.168.201.21@tcp [ 2046.320626] Key type .llcrypt unregistered [ 2046.322278] Key type ._llcrypt unregistered [ 2057.283741] Key type ._llcrypt registered [ 2057.284988] Key type .llcrypt registered [ 2058.397388] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:01:56 (1782482516) [ 2058.522842] LNet: 120579:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2058.522842] mask usage: [+|-] ... [ 2058.614374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.623568] alg: No test for adler32 (adler32-zlib) [ 2059.519652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.772455] LNet: 120663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.134500] Key type .llcrypt unregistered [ 2060.135660] Key type ._llcrypt unregistered [ 2064.528573] Key type ._llcrypt registered [ 2064.531780] Key type .llcrypt registered [ 2064.564634] LNet: 120892:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2064.564634] mask usage: [+|-] ... [ 2064.668338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2064.683247] alg: No test for adler32 (adler32-zlib) [ 2065.586249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2065.618133] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2071.343824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.121@tcp [ 2078.255923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.300748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2078.319460] LNet: Added LNI 192.168.201.21@tcp1 [8/256/0/180] [ 2079.030878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2079.461985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.121@tcp [ 2079.693712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2079.723264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2079.775325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2079.808608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2079.836899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2079.865663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2079.898912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2079.938714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2079.965682] LNet: 121314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2079.968512] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.121@tcp [ 2079.990163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2080.042699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2080.110206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2080.149645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2080.203821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2080.244580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2080.866483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2081.082524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2081.106139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2081.129451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2081.151878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2081.176589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2081.208887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2081.251503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2081.293524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2081.315941] LNet: 121431:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2081.335716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2081.373661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2081.400652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2081.438789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2081.465111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2081.490884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2081.997132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.121@tcp [ 2082.206718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2082.238639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2082.265788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2082.285794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2082.315083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2082.352682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2082.382291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2082.410087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2082.422582] LNet: 121549:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2082.440278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2082.487486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2082.520542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2082.539863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2082.571990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2082.593697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2083.096464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2083.320335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2083.351748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2083.373559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2083.390843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2083.419055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2083.444770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2083.482391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2083.509793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2083.536483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2083.563376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2083.593339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2083.620963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2083.645683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2083.671445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2087.773620] LNet: 121747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.782733] LNetError: 121747:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.794968] LNet: Removed LNI 192.168.201.21@tcp [ 2087.812970] LNet: Removed LNI 192.168.201.21@tcp1 [ 2088.173578] Key type .llcrypt unregistered [ 2088.176265] Key type ._llcrypt unregistered [ 2101.037563] Key type ._llcrypt registered [ 2101.039735] Key type .llcrypt registered [ 2101.985905] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:02:40 (1782482560) [ 2108.201647] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:02:46 (1782482566) [ 2113.237400] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:02:51 (1782482571) [ 2118.382505] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:02:56 (1782482576) [ 2122.997960] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:03:01 (1782482581) [ 2127.348444] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:03:05 (1782482585) [ 2132.239695] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:03:10 (1782482590) [ 2136.672260] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:03:15 (1782482595) [ 2140.876869] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:03:19 (1782482599) [ 2144.672369] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:03:23 (1782482603) [ 2149.088377] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:03:27 (1782482607) [ 2153.536495] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:03:32 (1782482612) [ 2157.377197] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:03:36 (1782482616) [ 2161.393945] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:03:39 (1782482619) [ 2166.235759] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:03:44 (1782482624) [ 2170.353452] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:03:49 (1782482629) [ 2174.472222] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:03:53 (1782482633) [ 2178.709675] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:03:57 (1782482637) [ 2178.953642] LNet: 132281:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2178.953642] mask usage: [+|-] ... [ 2179.054639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.069579] alg: No test for adler32 (adler32-zlib) [ 2179.938673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.019730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2180.037759] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2180.686988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.698696] LNet: 132452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.702533] LNetError: 132452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.714174] LNet: Removed LNI 192.168.201.21@tcp [ 2180.726856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.753629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2180.764909] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2185.005865] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:04:03 (1782482643) [ 2189.833450] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:04:08 (1782482648) [ 2194.295784] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:04:12 (1782482652) [ 2199.223902] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:04:17 (1782482657) [ 2203.083942] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:04:21 (1782482661) [ 2206.834911] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:04:25 (1782482665) [ 2210.798544] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:04:29 (1782482669) [ 2214.658984] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:04:33 (1782482673) [ 2218.456801] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:04:37 (1782482677) [ 2222.931125] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:04:41 (1782482681) [ 2226.803519] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:04:45 (1782482685) [ 2231.226735] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:04:49 (1782482689) [ 2235.632064] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1933 sec ========== 10:04:54 (1782482694) [ 2235.882743] LNet: 139910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.887552] LNetError: 139910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2235.895863] LNet: Removed LNI 192.168.201.21@tcp [ 2236.278398] Key type .llcrypt unregistered [ 2236.280217] Key type ._llcrypt unregistered [ 2243.511498] Key type ._llcrypt registered [ 2243.512640] Key type .llcrypt registered [ 2243.558106] LNet: 140367:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2243.558106] mask usage: [+|-] ... [ 2243.882333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2243.892312] alg: No test for adler32 (adler32-zlib) [ 2244.890788] Lustre: Lustre: Build Version: 2.17.54_98_g3a728b9 [ 2245.053437] LNet: Added LNI 192.168.201.21@tcp [8/256/0/180] [ 2246.703238] Key type lgssc registered [ 2247.664402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2291.772846] Lustre: Mounted lustre-client [ 2291.893203] LNet: 141853:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2291.893203] mask usage: [+|-] ... [ 2295.371547] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2299.238127] Lustre: Unmounted lustre-client [ 2334.001420] Key type lgssc unregistered [ 2334.227808] LNet: 142835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.233126] LNetError: 142835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.247874] LNet: Removed LNI 192.168.201.21@tcp [ 2334.769271] Key type .llcrypt unregistered [ 2334.771873] Key type ._llcrypt unregistered