[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451128028 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002265] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006361] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007010] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009088] LSM: Security Framework initializing [ 0.010025] Yama: becoming mindful. [ 0.011018] SELinux: Initializing. [ 0.012043] *** VALIDATE selinux *** [ 0.018622] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022506] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023085] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025069] *** VALIDATE tmpfs *** [ 0.026414] *** VALIDATE proc *** [ 0.027196] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029171] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030096] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032017] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.035984] debug: unmapping init [mem 0xffffffffb5259000-0xffffffffb5260fff] [ 0.037119] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.038519] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.039010] ... version: 2 [ 0.039797] ... bit width: 48 [ 0.040007] ... generic registers: 4 [ 0.040868] ... value mask: 0000ffffffffffff [ 0.041006] ... max period: 00007fffffffffff [ 0.042006] ... fixed-purpose events: 3 [ 0.042841] ... event mask: 000000070000000f [ 0.044122] rcu: Hierarchical SRCU implementation. [ 0.046061] smp: Bringing up secondary CPUs ... [ 0.047374] x86: Booting SMP configuration: [ 0.048014] .... node #0, CPUs: #1 #2 #3 [ 0.050800] smp: Brought up 1 node, 4 CPUs [ 0.052009] smpboot: Max logical packages: 1 [ 0.053009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.238014] node 0 deferred pages initialised in 184ms [ 0.241107] devtmpfs: initialized [ 0.242191] x86/mm: Memory block size: 128MB [ 0.245478] gcov: version magic: 0x41383552 [ 0.248230] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.252114] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.254245] pinctrl core: initialized pinctrl subsystem [ 0.256095] [ 0.256667] ************************************************************* [ 0.259012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.261010] ** ** [ 0.264013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.266008] ** ** [ 0.268010] ** This means that this kernel is built to expose internal ** [ 0.271007] ** IOMMU data structures, which may compromise security on ** [ 0.273008] ** your system. ** [ 0.275013] ** ** [ 0.278031] ** If you see this message and you are not debugging the ** [ 0.280018] ** kernel, report this immediately to your vendor! ** [ 0.282028] ** ** [ 0.284014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.287009] ************************************************************* [ 0.290139] NET: Registered protocol family 16 [ 0.291333] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.293030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.296318] cpuidle: using governor menu [ 0.298255] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.299350] PCI: Using configuration type 1 for base access [ 0.301089] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.307118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.309127] cryptd: max_cpu_qlen set to 1000 [ 0.311212] ACPI: Added _OSI(Module Device) [ 0.312009] ACPI: Added _OSI(Processor Device) [ 0.313009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.316360] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.323085] ACPI: Interpreter enabled [ 0.324045] ACPI: PM: (supports S0 S3 S4 S5) [ 0.326009] ACPI: Using IOAPIC for interrupt routing [ 0.327050] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.329232] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.336057] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.338018] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.341031] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.343887] acpiphp: Slot [2] registered [ 0.345069] acpiphp: Slot [5] registered [ 0.345925] acpiphp: Slot [6] registered [ 0.347064] acpiphp: Slot [3] registered [ 0.347949] acpiphp: Slot [4] registered [ 0.349089] acpiphp: Slot [7] registered [ 0.350085] acpiphp: Slot [8] registered [ 0.351095] acpiphp: Slot [9] registered [ 0.353062] acpiphp: Slot [10] registered [ 0.354074] acpiphp: Slot [11] registered [ 0.356076] acpiphp: Slot [12] registered [ 0.357072] acpiphp: Slot [13] registered [ 0.359051] acpiphp: Slot [14] registered [ 0.359958] acpiphp: Slot [15] registered [ 0.360059] acpiphp: Slot [16] registered [ 0.360915] acpiphp: Slot [17] registered [ 0.362045] acpiphp: Slot [18] registered [ 0.362878] acpiphp: Slot [19] registered [ 0.363042] acpiphp: Slot [20] registered [ 0.363882] acpiphp: Slot [21] registered [ 0.365065] acpiphp: Slot [22] registered [ 0.365919] acpiphp: Slot [23] registered [ 0.367045] acpiphp: Slot [24] registered [ 0.368062] acpiphp: Slot [25] registered [ 0.369093] acpiphp: Slot [26] registered [ 0.371100] acpiphp: Slot [27] registered [ 0.373085] acpiphp: Slot [28] registered [ 0.375051] acpiphp: Slot [29] registered [ 0.375933] acpiphp: Slot [30] registered [ 0.376045] acpiphp: Slot [31] registered [ 0.376826] PCI host bridge to bus 0000:00 [ 0.378010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.379013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.382015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.384009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.386015] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.387190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.388761] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.390819] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.396353] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.398428] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.400008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.402008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.403007] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.405356] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.406553] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.408019] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.409436] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411919] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.417639] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.421010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.425681] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.430007] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433008] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.449132] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.452835] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455578] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.473196] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.475316] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.478301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.481306] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.484165] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.490098] iommu: Default domain type: Passthrough [ 0.492384] SCSI subsystem initialized [ 0.494118] ACPI: bus type USB registered [ 0.495093] usbcore: registered new interface driver usbfs [ 0.497111] usbcore: registered new interface driver hub [ 0.499068] usbcore: registered new device driver usb [ 0.500116] pps_core: LinuxPPS API ver. 1 registered [ 0.501007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.503036] PTP clock support registered [ 0.505047] EDAC MC: Ver: 3.0.0 [ 0.506323] PCI: Using ACPI for IRQ routing [ 0.508593] NetLabel: Initializing [ 0.510010] NetLabel: domain hash size = 128 [ 0.511008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.513066] NetLabel: unlabeled traffic allowed by default [ 0.515095] vgaarb: loaded [ 0.516203] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.518011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.526267] clocksource: Switched to clocksource kvm-clock [ 0.614796] VFS: Disk quotas dquot_6.6.0 [ 0.616246] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.619147] *** VALIDATE ramfs *** [ 0.620879] *** VALIDATE hugetlbfs *** [ 0.622829] pnp: PnP ACPI init [ 0.625932] pnp: PnP ACPI: found 6 devices [ 0.640102] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.643444] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.645929] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.648079] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.650349] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.653530] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.657268] NET: Registered protocol family 2 [ 0.660273] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.664631] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.667919] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.673082] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.676029] TCP: Hash tables configured (established 65536 bind 65536) [ 0.678454] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.681158] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.683558] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.685963] NET: Registered protocol family 1 [ 0.687965] RPC: Registered named UNIX socket transport module. [ 0.689779] RPC: Registered udp transport module. [ 0.691241] RPC: Registered tcp transport module. [ 0.692752] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.694815] NET: Registered protocol family 44 [ 0.696213] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.698267] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.700310] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.702560] PCI: CLS 0 bytes, default 64 [ 0.704119] Unpacking initramfs... [ 2.089689] debug: unmapping init [mem 0xffff8d5d3cc64000-0xffff8d5d3ffcffff] [ 2.092910] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.094639] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.097041] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.616449] Initialise system trusted keyrings [ 2.617322] Key type blacklist registered [ 2.618548] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.627530] zbud: loaded [ 2.630278] *** VALIDATE nfs *** [ 2.631372] *** VALIDATE nfs4 *** [ 2.632714] pstore: using deflate compression [ 2.635048] Platform Keyring initialized [ 2.706317] NET: Registered protocol family 38 [ 2.708380] Key type asymmetric registered [ 2.710325] Asymmetric key parser 'x509' registered [ 2.712633] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.715555] io scheduler mq-deadline registered [ 2.717151] io scheduler kyber registered [ 2.718347] io scheduler bfq registered [ 2.719584] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.721568] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.723475] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.725366] ACPI: Power Button [PWRF] [ 2.730057] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.736336] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.748369] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.777114] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.803733] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.806864] Non-volatile memory driver v1.3 [ 2.808010] Linux agpgart interface v0.103 [ 2.827945] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.830299] vda: detected capacity change from 0 to 74706944 [ 2.839509] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.841236] vdb: detected capacity change from 0 to 1073741824 [ 2.844925] libphy: Fixed MDIO Bus: probed [ 2.848722] usbcore: registered new interface driver usbserial_generic [ 2.850186] usbserial: USB Serial support registered for generic [ 2.851511] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.854335] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.855862] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.857804] mousedev: PS/2 mouse device common for all mice [ 2.859894] rtc_cmos 00:05: RTC can wake from S4 [ 2.861672] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.862561] rtc_cmos 00:05: registered as rtc0 [ 2.866079] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.866596] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.869871] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.871445] intel_pstate: CPU model not supported [ 2.876343] hid: raw HID events driver (C) Jiri Kosina [ 2.878310] usbcore: registered new interface driver usbhid [ 2.880085] usbhid: USB HID core driver [ 2.881474] drop_monitor: Initializing network drop monitor service [ 2.883367] Initializing XFRM netlink socket [ 2.884436] NET: Registered protocol family 10 [ 2.886503] Segment Routing with IPv6 [ 2.887333] NET: Registered protocol family 17 [ 2.888529] mpls_gso: MPLS GSO support [ 2.892916] RAS: Correctable Errors collector initialized. [ 2.894926] AVX version of gcm_enc/dec engaged. [ 2.896647] AES CTR mode by8 optimization enabled [ 2.937646] sched_clock: Marking stable (2937625013, 0)->(3617011776, -679386763) [ 2.939994] registered taskstats version 1 [ 2.941186] Loading compiled-in X.509 certificates [ 2.943026] zswap: loaded using pool lzo/zbud [ 2.959198] Key type big_key registered [ 2.967925] Key type encrypted registered [ 2.969424] ima: No TPM chip found, activating TPM-bypass! [ 2.971287] ima: Allocated hash algorithm: sha1 [ 2.972789] ima: No architecture policies found [ 2.974351] evm: Initialising EVM extended attributes: [ 2.976087] evm: security.selinux [ 2.977213] evm: security.ima [ 2.978202] evm: security.capability [ 2.979430] evm: HMAC attrs: 0x1 [ 2.981503] rtc_cmos 00:05: setting system clock to 2026-08-17 04:05:02 UTC (1786939502) [ 2.987284] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 2.990021] debug: unmapping init [mem 0xffffffffb4f82000-0xffffffffb5258fff] [ 3.002073] Write protecting the kernel read-only data: 28672k [ 3.004817] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 3.006778] debug: unmapping init [mem 0xffffffffb3f14000-0xffffffffb3ffffff] [ 3.040924] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.049737] systemd[1]: Detected virtualization kvm. [ 3.051356] systemd[1]: Detected architecture x86-64. [ 3.052959] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.079354] systemd[1]: No hostname configured. [ 3.080860] systemd[1]: Set hostname to . [ 3.082514] random: systemd: uninitialized urandom read (16 bytes read) [ 3.084535] systemd[1]: Initializing machine ID from random generator. [ 3.130388] random: ln: uninitialized urandom read (6 bytes read) [ 3.201629] random: systemd: uninitialized urandom read (16 bytes read) [ 3.204415] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.208221] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.211836] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.782163] device-mapper: uevent: version 1.0.3 [ 3.784307] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.510958] virtio_net virtio0 ens2: renamed from eth0 [ 4.550577] scsi host0: ata_piix [ 4.573982] scsi host1: ata_piix [ 4.575443] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.577233] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.936718] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.350461] random: fast init done [ 9.491500] random: crng init done [ 9.492668] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.866070] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.042508] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.258915] SELinux: Disabled at runtime. [ 11.319791] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.328941] systemd[1]: Detected virtualization kvm. [ 11.330988] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.788040] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.791775] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.795254] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.799154] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.803350] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.814771] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.818434] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Huge Pages File System... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ 12.002560] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.322304] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.662146] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.738771] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.813518] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.822191] EDAC sbridge: Ver: 1.1.2 [ 13.768635] Key type dns_resolver registered [ 14.094813] NFS: Registering the id_resolver key type [ 14.096759] Key type id_resolver registered [ 14.098373] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg354-client login: [ 46.850139] libcfs: loading out-of-tree module taints kernel. [ 46.933335] Key type ._llcrypt registered [ 46.943304] Key type .llcrypt registered [ 47.147188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.153114] alg: No test for adler32 (adler32-zlib) [ 48.118831] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 48.397220] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 50.007167] Key type lgssc registered [ 50.639382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.379615] Lustre: Mounted lustre-client - version 2.17.57_2_gd3fb4ca [ 95.715348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 104.066911] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing check_logdir /tmp/testlogs/ [ 105.595198] Lustre: DEBUG MARKER: oleg354-client.virtnet: executing yml_node [ 107.185795] Lustre: DEBUG MARKER: Client: 2.17.57.2 [ 108.063159] Lustre: DEBUG MARKER: MDS: 2.17.57.2 [ 108.939662] Lustre: DEBUG MARKER: OSS: 2.17.57.2 [ 109.707405] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 17 00:06:48 EDT 2026 [ 119.263407] Lustre: lustre-OST0000-osc-ffff8d5d98741000: disconnect after 24s idle [ 119.472615] Lustre: Unmounted lustre-client [ 138.932187] Key type lgssc unregistered [ 139.201155] LNet: 8412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 139.217417] LNetError: 8412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 139.236688] LNet: Removed LNI 192.168.203.54@tcp [ 140.198155] Key type .llcrypt unregistered [ 140.204468] Key type ._llcrypt unregistered [ 154.656934] Key type ._llcrypt registered [ 154.658599] Key type .llcrypt registered [ 154.731066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 154.737836] alg: No test for adler32 (adler32-zlib) [ 155.545723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 155.575752] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 155.586127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 157.486233] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 157.522739] LNet: 9291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 157.525898] LNetError: 9291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 157.535290] LNet: Removed LNI 192.168.203.54@tcp [ 157.870570] Key type .llcrypt unregistered [ 157.873738] Key type ._llcrypt unregistered [ 160.990929] Key type ._llcrypt registered [ 160.992948] Key type .llcrypt registered [ 161.826170] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:07:40 (1786939660) [ 162.039818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 162.053396] alg: No test for adler32 (adler32-zlib) [ 162.896340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.950081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 162.963860] LNet: 9711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 162.976792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.003564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 163.016806] LNet: 9718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 166.716730] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:07:45 (1786939665) [ 166.771694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 166.799952] LNet: 10289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 166.825582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 166.882065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 170.243954] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:07:49 (1786939669) [ 170.354530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.364438] LNet: 10879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.380633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.416909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 173.459975] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:07:52 (1786939672) [ 173.501402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.517157] LNet: 11455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.530277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.570310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 176.635132] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:07:55 (1786939675) [ 176.721221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.737346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.772508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 176.944949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 180.215132] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:07:59 (1786939679) [ 180.264844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 180.279458] LNet: 12631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 180.283108] LNet: 12631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 180.295678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 180.328900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 180.523327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 180.717834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 183.798651] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:08:02 (1786939682) [ 183.851023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.873493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.925655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 186.998498] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:08:05 (1786939685) [ 187.077631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.101318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.135132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 187.155637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 187.345425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 187.370907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 187.601559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 187.628088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 187.840313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 187.864660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 188.082742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 188.100252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 188.287292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 188.305430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 188.516746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 188.534793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 188.730185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 188.754916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 188.958877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 188.978890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 189.193328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 189.213529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 189.392353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 189.420819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 189.631543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 189.653542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 189.852725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 189.871636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 193.174181] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:08:11 (1786939691) [ 193.246378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.264127] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.270920] LNet: 14492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 193.286867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.325832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 193.353584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 196.512332] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:08:15 (1786939695) [ 196.602604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.636901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.675793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 196.691369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 199.926777] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:08:18 (1786939698) [ 200.041147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.066625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.116676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 200.143326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 203.432561] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:08:22 (1786939702) [ 203.537694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.558233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.598494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 203.616163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 206.782177] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:08:25 (1786939705) [ 206.834784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.857050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.908117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 206.925770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 210.102765] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:08:29 (1786939709) [ 210.156170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.169917] LNet: 17418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 210.173780] LNet: 17418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 210.188963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.228326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 210.253358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 213.304483] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:08:32 (1786939712) [ 213.402838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.434688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.480063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 213.665727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 216.535257] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:08:35 (1786939715) [ 216.651385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.679425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.789413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 216.888918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 220.062053] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:08:38 (1786939718) [ 220.169943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.187120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.241801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 220.364069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 223.449823] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:08:42 (1786939722) [ 223.521779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.544349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.777087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 223.888765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 226.744111] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:08:45 (1786939725) [ 226.796830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.815152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.858611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 226.884985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 229.626283] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:08:48 (1786939728) [ 229.688497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.710698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.741030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 229.763923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 232.580967] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:08:51 (1786939731) [ 232.679257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 232.698063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 232.738724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 235.396747] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:08:54 (1786939734) [ 235.449392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.470067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.513717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 238.191711] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:08:57 (1786939737) [ 238.221782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.243960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.291625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 238.453341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 241.001294] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:09:00 (1786939740) [ 241.034154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.049906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.084602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 241.101345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 243.491352] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:09:02 (1786939742) [ 243.522360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.533184] LNet: 24025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 243.537804] LNet: 24025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 243.550831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.586719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 243.602816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 245.857233] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:09:04 (1786939744) [ 245.897972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.914677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.948388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 245.967877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 248.444747] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:09:07 (1786939747) [ 248.473494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 248.492380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.527516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 248.545255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 250.999321] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:09:10 (1786939750) [ 251.028783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.049357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.081421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 251.109827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 251.120186] LNetError: 25798:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 251.140188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 251.155546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 253.264908] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:09:12 (1786939752) [ 253.304131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.321654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.354170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EKFyfsy4cM [ 253.383107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EKFyfsy4cM [ 253.403374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.EKFyfsy4cM [ 255.709124] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:09:14 (1786939754) [ 255.774436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.791703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.816226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 255.833751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 255.855109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 258.064346] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:09:17 (1786939757) [ 258.109981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.129534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.162316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 258.174909] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 258.751976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 259.255357] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 260.969049] Key type lgssc registered [ 261.497429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 263.165409] LNetError: 27558:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.54@tcp99 has no available nets [ 264.401612] Key type lgssc unregistered [ 264.577580] LNetError: 28666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 264.588688] LNet: Removed LNI 192.168.203.54@tcp [ 264.869325] Key type .llcrypt unregistered [ 264.870838] Key type ._llcrypt unregistered [ 267.412184] Key type ._llcrypt registered [ 267.414194] Key type .llcrypt registered [ 267.956335] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:09:26 (1786939766) [ 268.141489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 268.148314] alg: No test for adler32 (adler32-zlib) [ 268.977636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.014959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 270.125287] LNet: Message forwarding will be enabled in 2s [ 273.183119] LNet: Message forwarding enabled [ 273.317325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 273.328143] LNet: Message forwarding will stop in 2s to 6s [ 277.565453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 277.979639] LNet: Message forwarding disabled [ 278.616481] LNet: Message forwarding will be enabled in 2s [ 281.719806] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:09:40 (1786939780) [ 281.814644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.826344] LNet: 30072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.449689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.485807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 285.090078] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:09:44 (1786939784) [ 285.142589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.153142] LNet: 30649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.165783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.190353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 285.206612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 285.221170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 285.236737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 285.252574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 285.271856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 285.287939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 285.307311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 285.327711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 285.344520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 285.361309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 285.375736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 285.390661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 285.406130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 285.423514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 285.440779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 285.457253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 285.474051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 285.491546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 285.503197] LNetError: 30696:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 287.916083] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:09:46 (1786939786) [ 287.987313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 287.999379] LNet: 31266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.010724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.047639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.774815] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:09:49 (1786939789) [ 291.566602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg354-server [ 292.364750] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:09:51 (1786939791) [ 292.449492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.462642] LNet: 32196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.473092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.497111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 292.509587] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 292.875536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[55-59]@tcp [ 292.887645] LNet: 32329:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 292.901482] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 292.910576] LNetError: 32200:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.55@tcp has gone from up to down [ 293.922588] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 293.928811] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 294.948496] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 294.958283] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 295.771864] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:09:54 (1786939794) [ 295.967570] LNetError: 32330:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.58@tcp has gone from up to down [ 295.982063] LNetError: 32330:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 296.613349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg354-server [ 296.994504] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 297.002856] LNetError: 32200:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 297.509963] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:09:56 (1786939796) [ 297.631557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.015235] LNetError: 32199:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.203.55@tcp [ 299.679596] LNet: 33264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.683429] LNetError: 33264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 299.694333] LNet: Removed LNI 192.168.203.54@tcp [ 299.705559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.736774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 299.753959] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 300.051518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[55-60/2]@tcp [ 300.063118] LNet: 33382:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 300.071969] LNet: 33382:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 300.083492] LNetError: 33269:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.59@tcp has gone from up to down [ 300.084882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[55-60/2]@tcp [ 300.092516] LNetError: 33269:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 302.839015] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:10:01 (1786939801) [ 302.901601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.913132] LNetError: 33973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.923062] LNet: Removed LNI 192.168.203.54@tcp [ 302.933129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.976197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 303.005852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.027096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.055968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.081537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.099258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.120507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 303.155367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.175861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.210784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.247956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.264879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.287870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 303.324850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.344923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.377575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.417943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.438431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.464762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 303.497838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.515967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.546634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.586130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.624732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.652467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 303.687459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.705759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.740299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 303.781565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.797582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.820158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 303.850027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.876373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.910680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 306.593040] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:10:05 (1786939805) [ 307.375782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg354-server [ 308.133103] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:10:07 (1786939807) [ 308.888595] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg354-server [ 309.593517] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:10:08 (1786939808) [ 309.687882] LNet: 35372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.691110] LNet: 35372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 309.978100] Key type .llcrypt unregistered [ 309.980334] Key type ._llcrypt unregistered [ 312.404234] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 312.443870] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 312.446114] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 312.448987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 314.679735] Key type ._llcrypt registered [ 314.681099] Key type .llcrypt registered [ 314.754059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 314.761453] alg: No test for adler32 (adler32-zlib) [ 315.578141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.605101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 315.618251] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 315.893661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 315.903197] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 316.708692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 316.722272] LNet: Removed LNI 10.1.2.3@tcp [ 316.767390] LNet: 36098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.770521] LNetError: 36098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.778287] LNet: Removed LNI 192.168.203.54@tcp [ 317.051431] Key type .llcrypt unregistered [ 317.053079] Key type ._llcrypt unregistered [ 321.195871] Key type ._llcrypt registered [ 321.197199] Key type .llcrypt registered [ 321.723894] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:10:20 (1786939820) [ 321.927365] Key type .llcrypt unregistered [ 321.928780] Key type ._llcrypt unregistered [ 324.131074] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 324.175893] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 324.178724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 324.181366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 326.399236] Key type ._llcrypt registered [ 326.401044] Key type .llcrypt registered [ 326.471907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.478989] alg: No test for adler32 (adler32-zlib) [ 327.290273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.315697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 327.328984] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 327.624817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 327.634623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 328.654673] LNet: 37595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 328.657876] LNetError: 37595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 328.664088] LNet: Removed LNI 10.1.2.3@tcp [ 328.672400] LNet: Removed LNI 192.168.203.54@tcp [ 328.940779] Key type .llcrypt unregistered [ 328.942265] Key type ._llcrypt unregistered [ 333.010456] Key type ._llcrypt registered [ 333.012223] Key type .llcrypt registered [ 333.547235] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:10:32 (1786939832) [ 333.746277] Key type .llcrypt unregistered [ 333.749357] Key type ._llcrypt unregistered [ 336.108926] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 336.162181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 336.164878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 336.167742] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 338.407275] Key type ._llcrypt registered [ 338.409103] Key type .llcrypt registered [ 338.495862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 338.502904] alg: No test for adler32 (adler32-zlib) [ 339.321264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.343328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 339.354986] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 340.190531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 340.202787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 341.077715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 341.087574] LNet: Removed LNI 10.1.2.3@tcp [ 341.095226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 341.119388] LNet: Removed LNI 10.1.2.31@tcp [ 341.128226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 341.141049] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 341.461440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 341.473729] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 341.870789] LNet: 39523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.875622] LNetError: 39523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 341.879932] LNet: Removed LNI 10.1.2.31@tcp [ 341.887344] LNet: Removed LNI 10.1.2.3@tcp [ 342.152614] Key type .llcrypt unregistered [ 342.154237] Key type ._llcrypt unregistered [ 346.481425] Key type ._llcrypt registered [ 346.482867] Key type .llcrypt registered [ 347.017845] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:10:45 (1786939845) [ 347.191915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 347.199104] alg: No test for adler32 (adler32-zlib) [ 348.010919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.036333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 348.047098] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 348.385163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.396178] LNet: 40480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.400468] LNetError: 40480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 348.412844] LNet: Removed LNI 192.168.203.54@tcp [ 348.424280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.446832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.54@tcp [ 348.460464] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 348.821372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.54@tcp [ 348.840193] LNet: Removed LNI 192.168.203.54@tcp [ 349.161947] Key type .llcrypt unregistered [ 349.164969] Key type ._llcrypt unregistered [ 353.694228] Key type ._llcrypt registered [ 353.695824] Key type .llcrypt registered [ 354.217668] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:10:53 (1786939853) [ 354.951053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg354-server [ 355.645432] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:10:54 (1786939854) [ 355.871523] Key type .llcrypt unregistered [ 355.874871] Key type ._llcrypt unregistered [ 358.195397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 358.236375] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 358.239743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 358.242170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 360.447895] Key type ._llcrypt registered [ 360.449436] Key type .llcrypt registered [ 360.517850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 360.524068] alg: No test for adler32 (adler32-zlib) [ 361.329903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.354560] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 361.641909] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 362.438922] LNet: 42456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.443409] LNetError: 42456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.446453] LNet: Removed LNI 192.168.203.54@tcp1 [ 362.452634] LNet: Removed LNI 10.1.2.3@tcp2 [ 362.706517] Key type .llcrypt unregistered [ 362.709374] Key type ._llcrypt unregistered [ 366.443743] Key type ._llcrypt registered [ 366.445008] Key type .llcrypt registered [ 366.899852] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:11:05 (1786939865) [ 367.017460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 367.023549] alg: No test for adler32 (adler32-zlib) [ 367.829480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.852515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 367.865705] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 368.159047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.168136] LNet: 43429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.171238] LNetError: 43429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 368.179293] LNet: Removed LNI 192.168.203.54@tcp [ 368.187397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.209266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 368.219256] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 370.373982] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:11:09 (1786939869) [ 370.406815] LNet: 44014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.408919] LNetError: 44014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 370.416406] LNet: Removed LNI 192.168.203.54@tcp [ 370.635589] Key type .llcrypt unregistered [ 370.637414] Key type ._llcrypt unregistered [ 372.487486] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 372.563558] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 372.565636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 372.567800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 374.735066] Key type ._llcrypt registered [ 374.736155] Key type .llcrypt registered [ 374.804870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 374.811490] alg: No test for adler32 (adler32-zlib) [ 375.615579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.639524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 375.651204] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 375.908237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 375.918704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 376.680797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.688139] LNet: 44759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.690439] LNetError: 44759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 376.692280] LNet: Removed LNI 10.1.2.3@tcp [ 376.707375] LNet: Removed LNI 192.168.203.54@tcp [ 376.715119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.741816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 376.753843] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 376.756836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 376.936300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.946134] LNet: Removed LNI 10.1.2.3@tcp [ 376.953358] LNet: Removed LNI 192.168.203.54@tcp [ 376.961045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.987663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 376.998616] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 377.000083] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 377.150229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 377.160543] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 377.525540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.537000] LNet: 44948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.540899] LNet: 44948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 377.544962] LNetError: 44948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 377.547427] LNetError: 44948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 377.549731] LNet: Removed LNI 10.1.2.3@tcp [ 377.550721] LNet: Removed LNI 192.168.203.54@tcp [ 377.557660] LNet: Removed LNI 192.168.203.54@tcp2 [ 377.566041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.591733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 377.604047] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 377.606522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 377.609524] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 379.903027] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:11:18 (1786939878) [ 379.960101] LNet: 45543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.963767] LNetError: 45543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 379.966570] LNetError: 45543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 379.970835] LNet: Removed LNI 10.1.2.3@tcp [ 379.972578] LNet: Removed LNI 192.168.203.54@tcp [ 379.979782] LNet: Removed LNI 192.168.203.54@tcp2 [ 380.230647] Key type .llcrypt unregistered [ 380.232709] Key type ._llcrypt unregistered [ 382.291726] Key type ._llcrypt registered [ 382.293336] Key type .llcrypt registered [ 382.372307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 382.379405] alg: No test for adler32 (adler32-zlib) [ 383.178568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.199197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 383.208195] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 383.524674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.533468] LNet: 45942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.536590] LNetError: 45942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.548364] LNet: Removed LNI 192.168.203.54@tcp [ 383.555820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.582285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 383.592889] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 383.768362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.784336] LNet: Removed LNI 192.168.203.54@tcp [ 383.792580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.818764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 383.828376] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 384.010681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.024733] LNet: Removed LNI 192.168.203.54@tcp [ 384.033889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.059270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 384.070337] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 384.228581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 384.281713] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 384.283728] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 384.285600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 386.440549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.448832] LNet: 46075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.451196] LNet: 46075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 386.454393] LNetError: 46075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 386.457257] LNetError: 46075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 386.463463] LNet: Removed LNI 192.168.203.54@tcp [ 386.471478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.497076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 386.507984] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 388.674266] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:11:27 (1786939887) [ 388.717600] LNet: 46667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.721357] LNetError: 46667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.730166] LNet: Removed LNI 192.168.203.54@tcp [ 388.963401] Key type .llcrypt unregistered [ 388.964624] Key type ._llcrypt unregistered [ 391.028568] Key type ._llcrypt registered [ 391.030168] Key type .llcrypt registered [ 391.101078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 391.108739] alg: No test for adler32 (adler32-zlib) [ 391.916669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.938521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 391.949412] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 392.272137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.280950] LNet: 47072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.284347] LNetError: 47072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 392.292312] LNet: Removed LNI 192.168.203.54@tcp [ 392.301753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.327172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 392.339148] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 394.635980] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:11:33 (1786939893) [ 394.695372] LNet: 47662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.699778] LNetError: 47662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 394.709636] LNet: Removed LNI 192.168.203.54@tcp [ 394.955504] Key type .llcrypt unregistered [ 394.957236] Key type ._llcrypt unregistered [ 397.079422] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 397.139843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 397.142549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 397.145277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 399.328651] Key type ._llcrypt registered [ 399.329813] Key type .llcrypt registered [ 399.393293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 399.398904] alg: No test for adler32 (adler32-zlib) [ 400.210080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.236290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 400.251104] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 400.610187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.621348] LNet: 48117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.625639] LNetError: 48117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 400.638587] LNet: Removed LNI 192.168.203.54@tcp [ 400.647681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.674713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 400.685788] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 402.847598] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:11:41 (1786939901) [ 402.898526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.911629] LNet: 48713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.915818] LNetError: 48713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.924440] LNet: Removed LNI 192.168.203.54@tcp [ 402.933966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.981407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 404.928518] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:11:44 (1786939904) [ 404.964246] LNet: 49289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.197415] Key type .llcrypt unregistered [ 405.199043] Key type ._llcrypt unregistered [ 407.213129] Key type ._llcrypt registered [ 407.214546] Key type .llcrypt registered [ 407.284665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 407.290913] alg: No test for adler32 (adler32-zlib) [ 408.094104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.119677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 408.130213] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 408.428731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.435848] LNet: 49696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.438627] LNetError: 49696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 408.447526] LNet: Removed LNI 192.168.203.54@tcp [ 408.456365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.482201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 408.492251] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 410.624268] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:11:49 (1786939909) [ 410.662410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.671561] LNet: 50284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.675934] LNetError: 50284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 410.684603] LNet: Removed LNI 192.168.203.54@tcp [ 410.694388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.722797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 412.913306] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:11:51 (1786939911) [ 413.442590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 413.957075] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:11:53 (1786939913) [ 414.484545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 414.986320] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:11:54 (1786939914) [ 415.494566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 416.053974] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:11:55 (1786939915) [ 416.619251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 417.213326] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:11:56 (1786939916) [ 417.750931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 418.336795] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:11:57 (1786939917) [ 418.367794] LNet: 52331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.621514] Key type .llcrypt unregistered [ 418.622996] Key type ._llcrypt unregistered [ 420.700427] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 420.753845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 420.755576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 420.757944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 422.970375] Key type ._llcrypt registered [ 422.971956] Key type .llcrypt registered [ 423.033689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 423.039857] alg: No test for adler32 (adler32-zlib) [ 423.838711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.866583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 423.874683] LNet: Added LNI 192.168.203.54@tcp [8/512/0/180] [ 423.876805] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 423.877937] LNet: 52660:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 423.907792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.915514] LNet: 52679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.918858] LNetError: 52679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 423.921146] LNet: Removed LNI 10.1.2.3@tcp [ 423.922287] LNetError: 52679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.929278] LNet: Removed LNI 192.168.203.54@tcp [ 423.936271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.960456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 423.969880] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 423.994819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.002164] LNet: 52703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.004219] LNetError: 52703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 424.011254] LNet: Removed LNI 192.168.203.54@tcp [ 424.019381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.045968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 424.055195] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 424.057424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 424.059107] LNet: 52711:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 424.093362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.101753] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.104979] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 424.108340] LNet: Removed LNI 10.1.2.3@tcp [ 424.109949] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 424.118280] LNet: Removed LNI 192.168.203.54@tcp [ 424.127969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.153803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 424.162859] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 424.165595] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 424.167145] LNet: 52737:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 424.234204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.242025] LNet: 52763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.245723] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 424.248762] LNet: Removed LNI 10.1.2.3@tcp [ 424.250074] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 424.261279] LNet: Removed LNI 192.168.203.54@tcp [ 424.269046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.293192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 424.303440] LNet: Added LNI 192.168.203.54@tcp [8/256/0/182] [ 424.305807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 424.340312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.348608] LNet: 52791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.352233] LNetError: 52791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 424.355561] LNet: Removed LNI 10.1.2.3@tcp [ 424.356463] LNetError: 52791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 424.366335] LNet: Removed LNI 192.168.203.54@tcp [ 424.375517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.400335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 424.409557] LNet: Added LNI 192.168.203.54@tcp [8/256/0/182] [ 424.411554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 424.413217] LNet: 52799:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 424.438457] LNet: Removed LNI 10.1.2.3@tcp [ 424.444252] LNet: Removed LNI 192.168.203.54@tcp [ 424.643698] Key type .llcrypt unregistered [ 424.644630] Key type ._llcrypt unregistered [ 428.232814] Key type ._llcrypt registered [ 428.233842] Key type .llcrypt registered [ 428.705915] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:12:07 (1786939927) [ 429.201904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 429.736834] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:12:08 (1786939928) [ 429.851361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 429.856410] alg: No test for adler32 (adler32-zlib) [ 430.658321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.679242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 430.687774] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 430.944303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.54 [ 430.952620] LNet: 54065:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 430.957419] LNet: Removed LNI 192.168.203.54@tcp [ 433.048048] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:12:12 (1786939932) [ 433.103592] LNet: 54633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.107094] LNetError: 54633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 433.115516] LNet: Removed LNI 192.168.203.54@tcp [ 433.331294] Key type .llcrypt unregistered [ 433.332656] Key type ._llcrypt unregistered [ 435.324067] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 435.424936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 436.669570] Key type ._llcrypt registered [ 436.670927] Key type .llcrypt registered [ 436.736775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 436.744556] alg: No test for adler32 (adler32-zlib) [ 437.691462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 439.709421] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:12:18 (1786939938) [ 439.738408] LNet: 55534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.741804] LNetError: 55534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 439.749579] LNet: Removed LNI 10.1.2.3@tcp [ 439.979377] Key type .llcrypt unregistered [ 439.981067] Key type ._llcrypt unregistered [ 441.933950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 442.025902] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 442.027519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 442.029884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 444.303324] Key type ._llcrypt registered [ 444.304944] Key type .llcrypt registered [ 444.369067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 444.374872] alg: No test for adler32 (adler32-zlib) [ 445.192739] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 447.242444] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:12:26 (1786939946) [ 447.724521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 448.241330] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:12:27 (1786939947) [ 448.272912] LNet: 56720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.275453] LNetError: 56720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 448.284452] LNet: Removed LNI 10.1.2.3@tcp [ 448.518778] Key type .llcrypt unregistered [ 448.520688] Key type ._llcrypt unregistered [ 450.580042] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 450.698793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 450.700362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 450.702307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 452.972641] Key type ._llcrypt registered [ 452.973649] Key type .llcrypt registered [ 453.035831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 453.041892] alg: No test for adler32 (adler32-zlib) [ 453.842558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.872620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 456.730483] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:12:35 (1786939955) [ 456.893224] LNet: 57959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.895898] LNetError: 57959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 456.902791] LNet: Removed LNI 10.1.2.3@tcp [ 457.155323] Key type .llcrypt unregistered [ 457.156618] Key type ._llcrypt unregistered [ 459.055987] Key type ._llcrypt registered [ 459.057144] Key type .llcrypt registered [ 459.118535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 459.123791] alg: No test for adler32 (adler32-zlib) [ 459.922237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 459.932381] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 462.837984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 463.043959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_interrupt -r 1 [ 463.059163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_interrupt -r 1 [ 463.074314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 463.083443] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 463.156949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 463.169794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 463.181901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 463.193553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 463.206390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 463.218445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 463.231059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 463.242979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 463.253624] LNet: 58399:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 463.262993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 463.276678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 463.289862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 463.302066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 463.314873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 463.328773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 463.341743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 463.514520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_dropped -r 1 [ 463.530434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_dropped -r 1 [ 463.545571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 463.626908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 463.639735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 463.651831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 463.664433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 463.677695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 463.693039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 463.708059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 463.721694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 463.738940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 463.754260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 463.770773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 463.785716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 463.800886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 463.814620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 463.830695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 464.013819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_aborted -r 1 [ 464.031615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_aborted -r 1 [ 464.050848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 464.058283] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 464.060923] LNet: Skipped 1 previous similar message [ 464.150562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 464.163935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 464.178447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 464.193278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 464.208034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 464.220742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 464.233636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 464.247884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 464.256918] LNet: 58617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 464.260641] LNet: 58617:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 464.269151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 464.282515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 464.294865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 464.306421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 464.318442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.330519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 464.343348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 464.520946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_no_route -r 1 [ 464.537962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_no_route -r 1 [ 464.553485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 464.633789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 464.645467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 464.657898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 464.671232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 464.683510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 464.697720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 464.713094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 464.728247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 464.744920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 464.759883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 464.774923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 464.790958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 464.806414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.819506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 464.832622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 465.019923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_timeout -r 1 [ 465.036143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_timeout -r 1 [ 465.054230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 465.064204] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 465.067170] LNet: Skipped 1 previous similar message [ 465.153626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 465.167363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 465.180920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 465.194089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 465.208097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 465.221167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 465.233837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 465.245929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 465.258959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 465.273058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 465.288777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 465.303328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 465.318206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 465.335234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 465.352200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 465.538577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_error -r 1 [ 465.554082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_error -r 1 [ 465.567600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 465.646371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 465.657908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 465.669355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 465.681526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 465.693289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 465.706974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 465.722598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 465.735198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 465.743966] LNet: 58944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 465.747569] LNet: 58944:0:(api-ni.c:296:recovery_interval_set()) Skipped 2 previous similar messages [ 465.755857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 465.767259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 465.780692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 465.793612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 465.808875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 465.824040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 465.837607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 465.930396] LNet: 58989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 465.932931] LNetError: 58989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 465.939295] LNet: Removed LNI 192.168.203.54@tcp [ 466.089483] Key type .llcrypt unregistered [ 466.090342] Key type ._llcrypt unregistered [ 471.624548] Key type ._llcrypt registered [ 471.625870] Key type .llcrypt registered [ 472.089967] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:12:51 (1786939971) [ 472.377284] Key type .llcrypt unregistered [ 472.378679] Key type ._llcrypt unregistered [ 474.322026] Key type ._llcrypt registered [ 474.323544] Key type .llcrypt registered [ 474.383158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 474.388066] alg: No test for adler32 (adler32-zlib) [ 475.186271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 475.196649] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 478.040148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 481.255803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.270355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 481.280371] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 481.715875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_interrupt -r 1 [ 481.729684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_interrupt -r 1 [ 481.746250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_interrupt -r 1 [ 481.762892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_interrupt -r 1 [ 481.778871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_interrupt -r 1 [ 481.793610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_interrupt -r 1 [ 481.810890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_interrupt -r 1 [ 481.827648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_interrupt -r 1 [ 481.844288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 481.854185] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 481.857513] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 481.934646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 481.947127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 481.960643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 481.974116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 481.987594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 482.001136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 482.014769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 482.030465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 482.040066] LNet: 60454:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 482.049282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 482.061621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 482.075632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 482.088872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 482.101685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.116081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 482.131283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 482.307541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_dropped -r 1 [ 482.322894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_dropped -r 1 [ 482.340269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_dropped -r 1 [ 482.356594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_dropped -r 1 [ 482.374177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_dropped -r 1 [ 482.389664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_dropped -r 1 [ 482.406106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_dropped -r 1 [ 482.423134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_dropped -r 1 [ 482.440646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 482.450897] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 482.454288] LNet: Skipped 2 previous similar messages [ 482.537221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 482.552668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 482.565714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 482.580433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 482.593241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 482.608384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 482.622100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 482.638218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 482.648677] LNet: 60575:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 482.657595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 482.671877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 482.685757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 482.699221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 482.715249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 482.729889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 482.746979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 482.921644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_aborted -r 1 [ 482.935855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_aborted -r 1 [ 482.950738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_aborted -r 1 [ 482.966783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_aborted -r 1 [ 482.980865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_aborted -r 1 [ 482.996273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_aborted -r 1 [ 483.010491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_aborted -r 1 [ 483.023079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_aborted -r 1 [ 483.036356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 483.112547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 483.125779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 483.138504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 483.149822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 483.161909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 483.172510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 483.185364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 483.196734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 483.210748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 483.225139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 483.237308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 483.249955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 483.263095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 483.276250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 483.290418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 483.454164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_no_route -r 1 [ 483.467933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_no_route -r 1 [ 483.480722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_no_route -r 1 [ 483.494065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_no_route -r 1 [ 483.508518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_no_route -r 1 [ 483.524042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_no_route -r 1 [ 483.538722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_no_route -r 1 [ 483.553065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_no_route -r 1 [ 483.565481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 483.572764] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 483.575332] LNet: Skipped 7 previous similar messages [ 483.648564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 483.660398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 483.673961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 483.686674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 483.699325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 483.712853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 483.724675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 483.739285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 483.747661] LNet: 60818:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 483.750323] LNet: 60818:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 483.757328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 483.768404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 483.778470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 483.789636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 483.802144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 483.816335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 483.828378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 483.990653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_timeout -r 1 [ 484.004551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_timeout -r 1 [ 484.018099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_timeout -r 1 [ 484.030786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_timeout -r 1 [ 484.043729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_timeout -r 1 [ 484.056761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_timeout -r 1 [ 484.069083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_timeout -r 1 [ 484.082285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_timeout -r 1 [ 484.095873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 484.169706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 484.181467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 484.195905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 484.209314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 484.220852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 484.233221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 484.248539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 484.262347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 484.273406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 484.283806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 484.294022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 484.303704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 484.313735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 484.323984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 484.336572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 484.496950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_error -r 1 [ 484.512207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_error -r 1 [ 484.524671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_error -r 1 [ 484.538609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_error -r 1 [ 484.552365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_error -r 1 [ 484.564139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_error -r 1 [ 484.575764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_error -r 1 [ 484.588989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_error -r 1 [ 484.601487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 484.672748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 484.682966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 484.695190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 484.706192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 484.717829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 484.730062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 484.742949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 484.756382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 484.768127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 484.781688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 484.794384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 484.806375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 484.817803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 484.831457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 484.843142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 486.992668] LNet: 61140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.996117] LNetError: 61140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.000902] LNet: Removed LNI 192.168.203.54@tcp [ 488.037292] LNet: Removed LNI 192.168.203.54@tcp1 [ 488.179684] Key type .llcrypt unregistered [ 488.181041] Key type ._llcrypt unregistered [ 493.158109] Key type ._llcrypt registered [ 493.159214] Key type .llcrypt registered [ 493.571961] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:13:12 (1786939992) [ 493.807390] Key type .llcrypt unregistered [ 493.808389] Key type ._llcrypt unregistered [ 495.604708] Key type ._llcrypt registered [ 495.606009] Key type .llcrypt registered [ 495.670120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.676657] alg: No test for adler32 (adler32-zlib) [ 496.473796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 496.483261] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 499.190137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 499.382343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_dropped -r 1 [ 499.395329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_dropped -r 1 [ 499.407907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 499.415062] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 499.486339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 499.497556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 499.507873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 499.517765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 499.526982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 499.536673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 499.547021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 499.558098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 499.564601] LNet: 62389:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 499.571024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 499.583450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 499.594508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 499.607447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 499.620522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 499.631888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 499.643398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 499.805807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_error -r 1 [ 499.819989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_error -r 1 [ 499.833335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 499.907463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 499.919641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 499.931812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 499.944694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 499.956714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 499.968543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 499.981612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 499.994449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 500.007319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 500.016795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 500.027456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 500.039079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 500.050619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.063142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 500.075912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 500.237707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_timeout -r 1 [ 500.252474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_timeout -r 1 [ 500.268227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 500.277540] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 500.279399] LNet: Skipped 1 previous similar message [ 500.344377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 500.356554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 500.368400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 500.380103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 500.392791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 500.403255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 500.413257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 500.427442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 500.435050] LNet: 62607:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 500.437237] LNet: 62607:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 500.444948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 500.455748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 500.467220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 500.479511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 500.491104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 500.504761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 500.516892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 500.608265] LNet: 62652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.611377] LNetError: 62652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.618261] LNet: Removed LNI 192.168.203.54@tcp [ 500.752323] Key type .llcrypt unregistered [ 500.753491] Key type ._llcrypt unregistered [ 505.761104] Key type ._llcrypt registered [ 505.762222] Key type .llcrypt registered [ 506.212726] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:13:25 (1786940005) [ 506.513526] Key type .llcrypt unregistered [ 506.515081] Key type ._llcrypt unregistered [ 508.499170] Key type ._llcrypt registered [ 508.500806] Key type .llcrypt registered [ 508.556020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 508.561993] alg: No test for adler32 (adler32-zlib) [ 509.359852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 509.368921] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 511.995750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 514.481974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.496449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 514.504228] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 514.978908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_dropped -r 1 [ 514.992945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e remote_dropped -r 1 [ 515.007238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_dropped -r 1 [ 515.021974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e remote_dropped -r 1 [ 515.033961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e remote_dropped -r 1 [ 515.049734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e remote_dropped -r 1 [ 515.067438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e remote_dropped -r 1 [ 515.082738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e remote_dropped -r 1 [ 515.098654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 515.108868] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 515.111473] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.154@tcp [ 515.188660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.200499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 515.212425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.224678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.235674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.251231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.265168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.278922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.287535] LNet: 64117:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 515.295217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.308413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.323121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.335641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.350204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.364535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.378590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 515.390580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.573477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_error -r 1 [ 515.589361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e remote_error -r 1 [ 515.604844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_error -r 1 [ 515.622476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e remote_error -r 1 [ 515.638871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e remote_error -r 1 [ 515.653582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e remote_error -r 1 [ 515.670995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e remote_error -r 1 [ 515.686701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e remote_error -r 1 [ 515.702646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 515.711840] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 515.713826] LNet: Skipped 2 previous similar messages [ 515.780370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.791742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 515.802611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.812132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.822614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.833946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.849400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.863187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.872837] LNet: 64240:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 515.882160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.894373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.904621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.916545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.931356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.943511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.954966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 515.967700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.148707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_timeout -r 1 [ 516.162515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e remote_timeout -r 1 [ 516.174694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_timeout -r 1 [ 516.187496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e remote_timeout -r 1 [ 516.200966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e remote_timeout -r 1 [ 516.214811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e remote_timeout -r 1 [ 516.228847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e remote_timeout -r 1 [ 516.241319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e remote_timeout -r 1 [ 516.253891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.154@tcp [ 516.324593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 516.336332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 516.349094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 516.361483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 516.374746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 516.390959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 516.404072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 516.416515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 516.431883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 516.444807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 516.457834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 516.469888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 516.481927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.495291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.510681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 516.523843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 518.596854] LNet: 64444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.599972] LNetError: 64444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.603124] LNet: Removed LNI 192.168.203.54@tcp [ 518.612367] LNet: Removed LNI 192.168.203.54@tcp1 [ 518.741599] Key type .llcrypt unregistered [ 518.742692] Key type ._llcrypt unregistered [ 523.538520] Key type ._llcrypt registered [ 523.539403] Key type .llcrypt registered [ 523.959922] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:13:43 (1786940023) [ 524.449824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 524.944756] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:13:44 (1786940024) [ 525.177179] Key type .llcrypt unregistered [ 525.178113] Key type ._llcrypt unregistered [ 526.948179] Key type ._llcrypt registered [ 526.949850] Key type .llcrypt registered [ 527.005818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 527.010852] alg: No test for adler32 (adler32-zlib) [ 527.815514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 527.825553] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 530.483544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 530.669541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e network_timeout -r 1 [ 530.683344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e network_timeout -r 1 [ 530.700417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 530.710302] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 530.778899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 530.793086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 530.805441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 530.816101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 530.828121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 530.840654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 530.849892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 530.859051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 530.864186] LNet: 65987:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 530.869244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 530.878554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 530.891362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 530.901116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 530.911174] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 530.914652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 530.929415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 531.022596] LNet: 66030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 531.026115] LNetError: 66030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 531.038249] LNet: Removed LNI 192.168.203.54@tcp [ 531.164400] Key type .llcrypt unregistered [ 531.165633] Key type ._llcrypt unregistered [ 536.290326] Key type ._llcrypt registered [ 536.291950] Key type .llcrypt registered [ 536.350034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.354373] alg: No test for adler32 (adler32-zlib) [ 537.151244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 537.161814] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 539.873675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 542.396182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.408747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 542.418311] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 542.837276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e network_timeout -r 1 [ 542.848749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e network_timeout -r 1 [ 542.860426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e network_timeout -r 1 [ 542.871771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e network_timeout -r 1 [ 542.885350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e network_timeout -r 1 [ 542.898960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e network_timeout -r 1 [ 542.914706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e network_timeout -r 1 [ 542.930304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e network_timeout -r 1 [ 542.945201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 542.955248] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 543.025741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 543.037705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 543.049042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 543.060096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 543.070441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 543.082273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 543.093767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 543.106437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 543.113048] LNet: 66910:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 543.119749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 543.131516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 543.143466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 543.153523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 543.165127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 543.178083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 545.193294] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 545.295547] LNet: 66988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.298087] LNetError: 66988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.301423] LNet: Removed LNI 192.168.203.54@tcp [ 545.309550] LNet: Removed LNI 192.168.203.54@tcp1 [ 545.433551] Key type .llcrypt unregistered [ 545.434886] Key type ._llcrypt unregistered [ 550.287169] Key type ._llcrypt registered [ 550.288337] Key type .llcrypt registered [ 550.693480] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:14:09 (1786940049) [ 550.799312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 550.804785] alg: No test for adler32 (adler32-zlib) [ 551.603065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.622606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 551.631714] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 551.866346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 551.873382] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 552.128028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 552.152327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 552.167986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 552.180555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 552.196286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 552.212586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 552.226848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.54@tcp [ 552.234993] LNetError: 68085:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.54@tcp to recovery [ 552.671210] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 552.671247] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 554.719255] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 555.373124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 558.815253] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 559.545314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 559.558831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 559.580336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.588066] LNet: 68203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.590428] LNetError: 68203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.592374] LNet: Removed LNI 192.168.203.54@tcp [ 559.599241] LNet: Removed LNI 192.168.203.54@tcp1 [ 559.605878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.623588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 559.631052] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 559.869967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 559.879735] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 560.152846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 560.167348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 560.190459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 560.206768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 560.220189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 560.232452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 560.242389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 560.254298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.54@tcp [ 560.262583] LNetError: 68475:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.54@tcp to recovery [ 562.719282] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 562.721379] LNet: Skipped 1 previous similar message [ 563.400777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 566.815370] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 571.686316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 571.698797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 571.712610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 573.821189] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:14:32 (1786940072) [ 573.848232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.855931] LNet: 69197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.858471] LNetError: 69197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.860486] LNetError: 69197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 573.863035] LNet: Removed LNI 192.168.203.54@tcp [ 573.869785] LNet: Removed LNI 192.168.203.54@tcp1 [ 573.880094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.903432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 573.913313] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 574.155287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 574.164632] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 574.473678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 574.499518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 574.510779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 574.525104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 574.538681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 574.555211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 574.569258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.54@tcp --health 0 [ 574.603061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 574.943290] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.54@tcp [ 575.967266] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 575.969931] LNet: Skipped 1 previous similar message [ 590.997393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 591.047727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 591.059780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 591.071445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.54@tcp --health 500 [ 591.108197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 598.331037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 598.354510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.365528] LNet: 69717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.369204] LNetError: 69717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.372527] LNetError: 69717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 598.376344] LNet: Removed LNI 192.168.203.54@tcp [ 598.385619] LNet: Removed LNI 192.168.203.54@tcp1 [ 598.394590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.420386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 598.432592] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 598.710863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 598.721221] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 599.043282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 599.059757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 599.084341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 599.097977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 599.112629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 599.127168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 599.141454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 599.156592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.54@tcp --health 0 [ 600.479243] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 600.481754] LNet: Skipped 3 previous similar messages [ 601.286122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 611.670241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 611.681840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 611.693891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 613.768547] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:15:12 (1786940112) [ 613.898555] LNet: 70784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.901214] LNetError: 70784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.904595] LNetError: 70784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 613.906781] LNet: Removed LNI 192.168.203.54@tcp [ 613.911316] LNet: Removed LNI 192.168.203.54@tcp1 [ 614.123902] Key type .llcrypt unregistered [ 614.125242] Key type ._llcrypt unregistered [ 616.097209] Key type ._llcrypt registered [ 616.098803] Key type .llcrypt registered [ 616.167172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 616.173444] alg: No test for adler32 (adler32-zlib) [ 616.970385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 616.980186] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 620.235248] Lustre: DEBUG MARKER: Initial discovery [ 620.255686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.154@tcp [ 620.984898] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 621.006835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 621.022210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.154@tcp [ 621.033745] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 621.043575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 621.056613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.154@tcp -r 1 -m GET -l 3 [ 621.071342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.154@tcp -r 1 -m GET -e local_error [ 621.085117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.154@tcp -r 1 -m PUT -l 6 [ 621.088074] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 621.537668] Lustre: DEBUG MARKER: Force oleg354-client.virtnet to discover 192.168.203.154@tcp (in background) [ 621.561351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.154@tcp [ 622.003102] Lustre: DEBUG MARKER: Force oleg354-server to discover 192.168.203.54@tcp [ 622.738166] Lustre: DEBUG MARKER: Wait for 71543 [ 624.607238] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 625.042400] Lustre: DEBUG MARKER: Finished wait on 71543 [ 625.059583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 625.074829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 625.168122] LNet: 72001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.170772] LNetError: 72001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.177254] LNet: Removed LNI 192.168.203.54@tcp [ 625.308488] Key type .llcrypt unregistered [ 625.309849] Key type ._llcrypt unregistered [ 630.369812] Key type ._llcrypt registered [ 630.370788] Key type .llcrypt registered [ 630.802645] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:15:29 (1786940129) [ 630.973432] Key type .llcrypt unregistered [ 630.974654] Key type ._llcrypt unregistered [ 632.913688] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 632.961194] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 632.963515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 632.965720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 635.161092] Key type ._llcrypt registered [ 635.162284] Key type .llcrypt registered [ 635.219450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 635.224866] alg: No test for adler32 (adler32-zlib) [ 636.020681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.037296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 636.044743] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 636.263675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 636.271754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 638.750633] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:15:37 (1786940137) [ 638.778730] LNet: 74070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.781391] LNetError: 74070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 638.783540] LNet: Removed LNI 10.1.2.3@tcp [ 638.789612] LNet: Removed LNI 192.168.203.54@tcp [ 638.994397] Key type .llcrypt unregistered [ 638.996127] Key type ._llcrypt unregistered [ 640.809148] Key type ._llcrypt registered [ 640.810554] Key type .llcrypt registered [ 640.864533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 640.869776] alg: No test for adler32 (adler32-zlib) [ 641.668875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.688681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 641.696801] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 641.916044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.923404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.236958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.54@tcp [ 643.295094] LNetError: Unexpected error -22 connecting to 192.168.203.54@tcp at host 192.168.203.54:988 [ 644.096786] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:15:43 (1786940143) [ 644.122876] LNet: 75205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.125283] LNetError: 75205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 644.128160] LNet: Removed LNI 10.1.2.3@tcp [ 644.135622] LNet: Removed LNI 192.168.203.54@tcp [ 644.349494] Key type .llcrypt unregistered [ 644.350660] Key type ._llcrypt unregistered [ 646.187977] Key type ._llcrypt registered [ 646.188866] Key type .llcrypt registered [ 646.245169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.250556] alg: No test for adler32 (adler32-zlib) [ 647.047051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.065696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 647.073074] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 647.286920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 647.294536] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 647.558582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.54@tcp1 --nid 192.168.203.54@tcp2 [ 648.797679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp1 192.168.203.54@tcp2 [ 648.804675] LNetError: 76050:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.54@tcp1 to 192.168.203.54@tcp2 - Route table may be misconfigured [ 648.812250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp2 192.168.203.54@tcp1 [ 650.650932] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:15:49 (1786940149) [ 650.696230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.707989] LNet: 76621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.710421] LNetError: 76621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.712989] LNet: Removed LNI 192.168.203.54@tcp1 [ 651.749485] LNet: Removed LNI 192.168.203.54@tcp2 [ 651.756596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.774777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.783349] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 652.005671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 652.014389] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 652.269913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 652.283291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.54@tcp [ 652.295113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp1 192.168.203.54@tcp1 [ 652.309469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.54@tcp -d 192.168.203.54@tcp -e network_timeout [ 652.321697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -e network_timeout [ 652.332770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -e network_timeout [ 652.345490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -e network_timeout [ 652.359624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 652.369553] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 652.376240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 652.405494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 652.437896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 652.767051] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 654.265279] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:15:53 (1786940153) [ 654.291660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.299575] LNet: 77478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.301803] LNetError: 77478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.304987] LNetError: 77478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 654.308110] LNet: Removed LNI 192.168.203.54@tcp [ 655.334331] LNet: Removed LNI 192.168.203.54@tcp1 [ 655.342942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.367411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 655.455524] LNet: 77525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.595069] Key type .llcrypt unregistered [ 655.595974] Key type ._llcrypt unregistered [ 658.860785] Key type ._llcrypt registered [ 658.862146] Key type .llcrypt registered [ 659.278816] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:15:58 (1786940158) [ 659.389551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.395086] alg: No test for adler32 (adler32-zlib) [ 660.206722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 660.633336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 661.120873] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:16:00 (1786940160) [ 661.144991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.152721] LNet: 78632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.159870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.179729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.187122] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 661.395859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 661.402100] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 661.658633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 661.672425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp1 [ 661.683197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp1 [ 663.497129] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:16:02 (1786940162) [ 664.019505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg354-server [ 664.564949] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:16:03 (1786940163) [ 665.072637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg354-server [ 665.530365] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:16:04 (1786940164) [ 666.098523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg354-server [ 666.613304] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:16:05 (1786940165) [ 667.172748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg354-server [ 667.680719] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:16:06 (1786940166) [ 668.233018] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg354-server [ 668.736116] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:16:07 (1786940167) [ 669.284192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg354-server [ 669.811457] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:16:08 (1786940168) [ 670.358510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg354-server [ 670.897241] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:16:09 (1786940169) [ 671.448249] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg354-server [ 671.930322] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:16:11 (1786940171) [ 672.468678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg354-server [ 673.014497] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:16:12 (1786940172) [ 673.036733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 673.045332] LNet: 82504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.048268] LNetError: 82504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.051127] LNet: Removed LNI 192.168.203.54@tcp [ 674.085429] LNet: Removed LNI 192.168.203.54@tcp1 [ 674.093095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.113678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 674.122178] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 674.345419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 674.371904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 674.423394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.433272] LNet: 82657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.437226] LNetError: 82657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.440769] LNetError: 82657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 674.450294] LNet: Removed LNI 192.168.203.54@tcp [ 674.458220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.476771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 674.485835] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 674.715939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 674.770779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.813312] LNet: Removed LNI 192.168.203.54@tcp [ 675.820984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.842031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.851886] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 676.095466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 676.119532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 676.181495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.192466] LNetError: 82949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.196340] LNetError: 82949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 676.207394] LNet: Removed LNI 192.168.203.54@tcp [ 676.215283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.234248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 676.246239] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 676.502207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 676.563753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.572159] LNet: 83094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.574143] LNet: 83094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 677.605420] LNet: Removed LNI 192.168.203.54@tcp [ 677.612403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.631196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.638483] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 677.860848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 677.882070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 677.938738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.013258] LNet: Removed LNI 192.168.203.54@tcp [ 679.020853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.037592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 679.045732] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 679.281810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 679.346895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.353498] LNetError: 83387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.355696] LNetError: 83387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 679.362258] LNet: Removed LNI 192.168.203.54@tcp [ 679.368546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.384515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.391850] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 679.611680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 679.640720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 679.702914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.741256] LNet: Removed LNI 192.168.203.54@tcp [ 680.748108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.764894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 680.772338] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 681.022433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 681.089513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.098057] LNet: 83679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.099827] LNet: 83679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 682.149269] LNet: Removed LNI 192.168.203.54@tcp [ 682.155553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.174940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.184449] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 682.448451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 682.478641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 682.551308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.621344] LNet: Removed LNI 192.168.203.54@tcp [ 683.628513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.648566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 683.657146] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 683.883129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 683.953785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.962722] LNetError: 83971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.965269] LNetError: 83971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 683.973378] LNet: Removed LNI 192.168.203.54@tcp [ 683.981565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.998697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.006548] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 684.222831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 684.253788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 684.329547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.349317] LNet: Removed LNI 192.168.203.54@tcp [ 685.356481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.374334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 685.382818] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 685.622495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 685.701172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.757311] LNet: Removed LNI 192.168.203.54@tcp [ 686.764104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.782523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 686.791708] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 687.013883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 687.035746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.052395] LNet: Removed LNI 192.168.203.54@tcp [ 687.059611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.077039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 687.085295] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 687.345635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 687.375906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.396515] LNet: Removed LNI 192.168.203.54@tcp [ 687.404750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.422757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 687.456044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.470107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.489679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 687.516057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.530916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.547818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 687.570892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.582588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.600887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 687.609096] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 687.836372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.850261] LNet: Removed LNI 192.168.203.54@tcp [ 687.856784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.879856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.927078] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:16:29 (1786940189) [ 689.977717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.986491] LNet: 85283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.989141] LNet: 85283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 689.997405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.016544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 690.025900] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/180] [ 690.375875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.390272] LNet: Removed LNI 192.168.203.54@tcp231 [ 690.396533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.419615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 690.429331] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/180] [ 690.432193] LNet: 85429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 690.557082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 690.575393] LNet: Removed LNI 192.168.203.54@tcp231 [ 690.581215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 690.589655] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/180] [ 690.920728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.934272] LNet: Removed LNI 192.168.203.54@tcp231 [ 690.940531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.962634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 690.973662] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/0] [ 690.975471] LNet: 85581:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 691.097897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 691.114852] LNet: Removed LNI 192.168.203.54@tcp231 [ 691.121904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 691.130551] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/0] [ 691.465674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.484323] LNet: Removed LNI 192.168.203.54@tcp231 [ 691.491131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.512689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 691.522503] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/60] [ 691.641247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 691.660294] LNet: Removed LNI 192.168.203.54@tcp231 [ 691.666347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 691.674549] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/60] [ 692.003892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.010540] LNetError: 85876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.012778] LNetError: 85876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 692.021350] LNet: Removed LNI 192.168.203.54@tcp231 [ 692.027576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.050843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 692.059462] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/180] [ 692.061974] LNet: 85884:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 692.064343] LNet: 85884:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 692.179964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 692.196304] LNet: Removed LNI 192.168.203.54@tcp231 [ 692.202705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 692.211616] LNet: Added LNI 192.168.203.54@tcp231 [8/256/0/180] [ 694.398486] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:16:33 (1786940193) [ 694.435654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.452596] LNet: Removed LNI 192.168.203.54@tcp231 [ 694.462768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.485343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 694.493712] LNet: Added LNI 192.168.203.54@tcp232 [8/256/0/180] [ 694.708394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 694.721626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 694.738453] LNet: Removed LNI 192.168.203.54@tcp232 [ 694.744644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 694.753404] LNet: Added LNI 192.168.203.54@tcp232 [8/256/0/180] [ 694.981616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 694.997293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.016278] LNet: Removed LNI 192.168.203.54@tcp232 [ 695.022439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 695.031045] LNet: Added LNI 192.168.203.54@tcp232 [8/256/0/180] [ 695.257745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 695.272398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.285618] LNet: Removed LNI 192.168.203.54@tcp232 [ 695.291661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 695.300770] LNet: Added LNI 192.168.203.54@tcp232 [8/256/0/180] [ 695.526731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 695.539308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 695.556311] LNet: Removed LNI 192.168.203.54@tcp232 [ 695.562989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 697.405794] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:16:36 (1786940196) [ 697.733048] Key type .llcrypt unregistered [ 697.733880] Key type ._llcrypt unregistered [ 699.484857] Key type ._llcrypt registered [ 699.485898] Key type .llcrypt registered [ 699.541804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.547038] alg: No test for adler32 (adler32-zlib) [ 700.342746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 700.352829] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 703.060038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 705.880282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.895713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 705.906052] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 706.410508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_interrupt -r 2 [ 706.427232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_interrupt -r 2 [ 706.443408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_interrupt -r 2 [ 706.459828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_interrupt -r 2 [ 706.477856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_interrupt -r 2 [ 706.499170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_interrupt -r 2 [ 706.516655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_interrupt -r 2 [ 706.534122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_interrupt -r 2 [ 706.550974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 706.559747] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 706.562488] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 706.657768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 706.670392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 706.682956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 706.694657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 706.708912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 706.720780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 706.733319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 706.745400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 706.752086] LNet: 88384:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 706.758738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 706.771753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 706.785164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 706.797615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 706.811111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 706.825115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 707.002360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_dropped -r 2 [ 707.018719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_dropped -r 2 [ 707.034942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_dropped -r 2 [ 707.048633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_dropped -r 2 [ 707.063289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_dropped -r 2 [ 707.078525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_dropped -r 2 [ 707.093347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_dropped -r 2 [ 707.108949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_dropped -r 2 [ 707.126440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 707.135612] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 707.219227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 707.232922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 707.244829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 707.255273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 707.265360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 707.277534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 707.289406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 707.300657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 707.307696] LNet: 88504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 707.315104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 707.327600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 707.338705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 707.350710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 707.361658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 707.373166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 707.545586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_aborted -r 2 [ 707.559832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_aborted -r 2 [ 707.573207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_aborted -r 2 [ 707.588510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_aborted -r 2 [ 707.604449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_aborted -r 2 [ 707.620230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_aborted -r 2 [ 707.636089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_aborted -r 2 [ 707.649098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_aborted -r 2 [ 707.662345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 707.747833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 707.760719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 707.775548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 707.787504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 707.801873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 707.814875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 707.826409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 707.837438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 707.848391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 707.858867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 707.869676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 707.880647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 707.892465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 707.906515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 708.080760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_no_route -r 2 [ 708.093766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_no_route -r 2 [ 708.108613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_no_route -r 2 [ 708.122424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_no_route -r 2 [ 708.136986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_no_route -r 2 [ 708.151337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_no_route -r 2 [ 708.166191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_no_route -r 2 [ 708.185076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_no_route -r 2 [ 708.201640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 708.210290] LNet: There was an unexpected network error while writing to 192.168.203.154: rc = -22 [ 708.212847] LNet: Skipped 2 previous similar messages [ 708.296644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 708.308622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 708.320276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 708.332738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 708.346899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 708.360642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 708.372685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 708.385851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 708.395494] LNet: 88744:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 708.397827] LNet: 88744:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 708.404994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 708.416672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 708.428589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 708.442546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 708.455804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 708.469518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 708.645469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e local_timeout -r 2 [ 708.659641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e local_timeout -r 2 [ 708.672641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e local_timeout -r 2 [ 708.688081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e local_timeout -r 2 [ 708.703838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e local_timeout -r 2 [ 708.720712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e local_timeout -r 2 [ 708.734697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e local_timeout -r 2 [ 708.750588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e local_timeout -r 2 [ 708.764001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 708.856334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 708.869096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 708.883187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 708.895705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 708.906514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 708.917377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 708.929590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 708.940579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 708.952084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 708.963486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 708.976000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 708.987642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 709.002760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 709.015784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 709.196955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp -m GET -e remote_dropped -r 2 [ 709.214720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.154@tcp1 -m GET -e remote_dropped -r 2 [ 709.229458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp -m GET -e remote_dropped -r 2 [ 709.242993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp -d 192.168.203.54@tcp1 -m GET -e remote_dropped -r 2 [ 709.257883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp -m GET -e remote_dropped -r 2 [ 709.272425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.154@tcp1 -m GET -e remote_dropped -r 2 [ 709.286918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp -m GET -e remote_dropped -r 2 [ 709.301577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.54@tcp1 -d 192.168.203.54@tcp1 -m GET -e remote_dropped -r 2 [ 709.316864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.54@tcp 192.168.203.154@tcp [ 709.326898] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.154@tcp [ 709.401778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 709.415365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 709.427264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 709.439420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 709.451458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 709.466513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 709.480973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 709.494747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 709.511219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 709.524905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 709.536485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 709.547085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 709.559811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 709.573198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 711.899986] LNet: 89062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.902279] LNetError: 89062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.904977] LNet: Removed LNI 192.168.203.54@tcp [ 711.915310] LNet: Removed LNI 192.168.203.54@tcp1 [ 712.056531] Key type .llcrypt unregistered [ 712.057599] Key type ._llcrypt unregistered [ 717.594202] Key type ._llcrypt registered [ 717.595733] Key type .llcrypt registered [ 718.002261] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:16:57 (1786940217) [ 718.127279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 718.133654] alg: No test for adler32 (adler32-zlib) [ 718.930901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.953615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.963656] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 719.214693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 719.225280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.54@tcp [ 719.232609] LNetError: 90038:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.54@tcp to recovery [ 719.261060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.54@tcp [ 719.290934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 721.154355] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:17:00 (1786940220) [ 721.187731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.197605] LNet: 90623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.200799] LNetError: 90623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.207612] LNet: Removed LNI 192.168.203.54@tcp [ 721.217455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.235802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.244331] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 721.474097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 721.481598] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 721.796133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.54@tcp [ 721.803600] LNetError: 90893:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.54@tcp to recovery [ 722.271300] LNet: 1 local NIs in recovery (showing 1): 192.168.203.54@tcp [ 722.271819] LNet: There was an unexpected network error while writing to 192.168.203.54: rc = -22 [ 722.872701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.54@tcp [ 726.934910] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:17:06 (1786940226) [ 727.478684] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg354-server [ 727.990617] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:17:07 (1786940227) [ 728.538153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg354-server [ 729.022093] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:17:08 (1786940228) [ 729.569903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg354-server [ 730.060152] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:17:09 (1786940229) [ 730.090533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.099050] LNet: 92515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.103758] LNetError: 92515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.107703] LNet: Removed LNI 192.168.203.54@tcp [ 730.114884] LNet: Removed LNI 192.168.203.54@tcp1 [ 730.125802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.144745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 730.153080] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 730.385034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 730.405637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 730.421690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 730.448882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 730.464499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 730.481731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 732.464653] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:17:11 (1786940231) [ 732.493882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.502485] LNet: 93239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.505299] LNetError: 93239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.508513] LNetError: 93239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 732.515559] LNet: Removed LNI 192.168.203.54@tcp [ 732.525171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.546323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 732.554802] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 734.793436] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:17:13 (1786940233) [ 734.827054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.835089] LNet: 93944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.837286] LNetError: 93944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.848118] LNet: Removed LNI 192.168.203.54@tcp [ 734.856445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.877379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.887944] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 737.075574] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:17:16 (1786940236) [ 737.516728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 738.014990] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:17:17 (1786940237) [ 738.181320] LNet: Removed LNI 192.168.203.54@tcp [ 738.383520] Key type .llcrypt unregistered [ 738.384425] Key type ._llcrypt unregistered [ 740.215219] Key type ._llcrypt registered [ 740.216874] Key type .llcrypt registered [ 740.285812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.292704] alg: No test for adler32 (adler32-zlib) [ 741.096623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 741.108218] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 743.872924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 745.276230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.154@tcp [ 745.377382] LNet: 95404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.379456] LNetError: 95404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.391289] LNet: Removed LNI 192.168.203.54@tcp [ 745.510249] Key type .llcrypt unregistered [ 745.511522] Key type ._llcrypt unregistered [ 749.088640] Key type ._llcrypt registered [ 749.089885] Key type .llcrypt registered [ 749.553220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 750.020435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 750.529402] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:17:29 (1786940249) [ 751.005497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 751.482573] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:17:30 (1786940250) [ 752.014494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg354-server [ 752.520272] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:17:31 (1786940251) [ 753.075960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg354-server [ 753.599267] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:17:32 (1786940252) [ 753.965949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 754.006880] alg: No test for adler32 (adler32-zlib) [ 754.857727] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 754.958476] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 756.543166] Key type lgssc registered [ 756.865530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 764.112232] Key type lgssc unregistered [ 764.224394] LNet: 98575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.227373] LNetError: 98575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.236278] LNet: Removed LNI 192.168.203.54@tcp [ 764.474392] Key type .llcrypt unregistered [ 764.475468] Key type ._llcrypt unregistered [ 768.947963] Key type ._llcrypt registered [ 768.948898] Key type .llcrypt registered [ 769.365084] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:17:48 (1786940268) [ 769.464646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 769.469391] alg: No test for adler32 (adler32-zlib) [ 770.264132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.284962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 770.299345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 770.312248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 770.326479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 772.246702] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:17:51 (1786940271) [ 772.275689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.284564] LNet: 99981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.292847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.317772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.326370] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 772.600473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.607128] LNetError: 100130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.619216] LNet: Removed LNI 192.168.203.54@tcp [ 772.625239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.644859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 772.654254] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 772.797991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 772.805498] LNet: 100159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.807301] LNet: 100159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 772.816250] LNet: Removed LNI 192.168.203.54@tcp [ 772.822912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.840764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 772.849110] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 774.880799] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:17:53 (1786940273) [ 774.907579] LNet: 100743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.910533] LNetError: 100743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.913036] LNetError: 100743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 774.926769] LNet: Removed LNI 192.168.203.54@tcp [ 775.134523] Key type .llcrypt unregistered [ 775.135792] Key type ._llcrypt unregistered [ 776.956737] Key type ._llcrypt registered [ 776.958393] Key type .llcrypt registered [ 777.017624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.021968] alg: No test for adler32 (adler32-zlib) [ 777.821834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.844295] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 777.845546] LNet: *** cfs_fail_loc=e003, val=100*** [ 778.079582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 778.088778] Ping reply received with mapping: [ 778.089792] found metadata reply NI[192.168.203.54@tcp] [ 778.091065] 36 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 778.092313] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 778.093550] 5a 5a 5a 5a 5a 5a 5a 5a [ 778.134829] LNet: 101149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.137322] LNetError: 101149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.147332] LNet: Removed LNI 192.168.203.54@tcp [ 778.365395] Key type .llcrypt unregistered [ 778.366625] Key type ._llcrypt unregistered [ 781.700521] Key type ._llcrypt registered [ 781.701501] Key type .llcrypt registered [ 782.132557] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:18:01 (1786940281) [ 782.287311] Key type .llcrypt unregistered [ 782.288359] Key type ._llcrypt unregistered [ 784.028624] Key type ._llcrypt registered [ 784.029635] Key type .llcrypt registered [ 784.091886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.098372] alg: No test for adler32 (adler32-zlib) [ 784.972802] LNetError: 102205:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 785.092572] Key type .llcrypt unregistered [ 785.093509] Key type ._llcrypt unregistered [ 786.575562] Key type ._llcrypt registered [ 786.576912] Key type .llcrypt registered [ 786.968068] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:18:06 (1786940286) [ 787.123233] Key type .llcrypt unregistered [ 787.124209] Key type ._llcrypt unregistered [ 789.014119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 789.017126] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.186447] Key type ._llcrypt registered [ 791.187715] Key type .llcrypt registered [ 791.257313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 791.263706] alg: No test for adler32 (adler32-zlib) [ 792.066323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.090873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 796.012426] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:18:15 (1786940295) [ 796.051371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.060940] LNet: 103987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.064340] LNetError: 103987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 796.078396] LNet: Removed LNI 10.1.2.3@tcp [ 796.089751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.350608] Key type .llcrypt unregistered [ 796.351846] Key type ._llcrypt unregistered [ 798.520496] Key type ._llcrypt registered [ 798.522275] Key type .llcrypt registered [ 798.580274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.586185] alg: No test for adler32 (adler32-zlib) [ 799.396208] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 799.423841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 799.481610] LNet: 104283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.484262] LNetError: 104283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.495377] LNet: Removed LNI 192.168.203.54@tcp [ 799.717543] Key type .llcrypt unregistered [ 799.718963] Key type ._llcrypt unregistered [ 803.427836] Key type ._llcrypt registered [ 803.428925] Key type .llcrypt registered [ 803.910032] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:18:22 (1786940302) [ 804.156281] Key type .llcrypt unregistered [ 804.157342] Key type ._llcrypt unregistered [ 806.178825] Key type ._llcrypt registered [ 806.179963] Key type .llcrypt registered [ 806.252224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.258329] alg: No test for adler32 (adler32-zlib) [ 809.216625] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:18:28 (1786940308) [ 809.248905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.261826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.291392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.299840] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 809.529339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.540467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 809.556620] LNet: Removed LNI 192.168.203.54@tcp [ 809.645447] LNet: 106035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.777318] Key type .llcrypt unregistered [ 809.778554] Key type ._llcrypt unregistered [ 813.448277] Key type ._llcrypt registered [ 813.450041] Key type .llcrypt registered [ 813.886792] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:18:32 (1786940312) [ 814.002689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.008785] alg: No test for adler32 (adler32-zlib) [ 814.805063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.823225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.831764] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 815.074346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 815.086845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.54@tcp [ 815.093525] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 815.095140] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 815.096643] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 815, 0, 0 [ 815.098633] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 815.099819] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 815.101418] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 815.102795] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 815, 0, 0 [ 815.104587] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 815.105629] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 815.107014] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 815.108357] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 815, 0, 865 [ 815.110120] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 815.111091] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 815.112514] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 815.113852] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 815, 0, 865 [ 815.115814] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 817.114409] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:18:36 (1786940316) [ 817.248460] LNet: 107573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.251389] LNetError: 107573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.258554] LNet: Removed LNI 192.168.203.54@tcp [ 817.478478] Key type .llcrypt unregistered [ 817.479708] Key type ._llcrypt unregistered [ 819.427503] Key type ._llcrypt registered [ 819.428569] Key type .llcrypt registered [ 819.490126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.495420] alg: No test for adler32 (adler32-zlib) [ 820.293708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 820.302689] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 823.080376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 825.801675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.814221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 825.822855] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 826.133744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.162676] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.165328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.167874] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.315421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 828.323493] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 831.618108] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:18:50 (1786940330) [ 831.649203] LNet: 109074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.651935] LNetError: 109074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.655787] LNet: Removed LNI 192.168.203.54@tcp [ 832.671178] LNet: Removed LNI 192.168.203.54@tcp1 [ 832.673110] LNetError: 109074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.677120] LNetError: 109074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 832.686516] LNet: Removed LNI 10.1.2.3@tcp99 [ 832.908393] Key type .llcrypt unregistered [ 832.909465] Key type ._llcrypt unregistered [ 834.854116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 834.855623] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.032703] Key type ._llcrypt registered [ 837.033838] Key type .llcrypt registered [ 837.097274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.104417] alg: No test for adler32 (adler32-zlib) [ 837.915470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.936881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.947741] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 838.227227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.238879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.048544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 839.066554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 839.084183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 839.093325] LNetError: 109827:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 839.107478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 839.126640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 839.278484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 839.308961] LNet: 109847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.311589] LNetError: 109847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 839.313940] LNet: Removed LNI 10.1.2.3@tcp [ 840.357694] LNet: Removed LNI 192.168.203.54@tcp [ 840.606472] Key type .llcrypt unregistered [ 840.607675] Key type ._llcrypt unregistered [ 844.466869] Key type ._llcrypt registered [ 844.468196] Key type .llcrypt registered [ 844.963423] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:19:03 (1786940343) [ 845.101283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.106804] alg: No test for adler32 (adler32-zlib) [ 845.905390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.921845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.929300] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 846.164419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 846.190177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg354-client@tcp [ 848.372562] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:19:07 (1786940347) [ 848.404623] LNet: 111376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.407757] LNetError: 111376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.416129] LNet: Removed LNI 192.168.203.54@tcp [ 848.662515] Key type .llcrypt unregistered [ 848.664479] Key type ._llcrypt unregistered [ 850.713880] Key type ._llcrypt registered [ 850.715155] Key type .llcrypt registered [ 850.784885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.790535] alg: No test for adler32 (adler32-zlib) [ 851.585762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 851.594524] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 851.608725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 853.741202] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:19:12 (1786940352) [ 854.304575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 854.898151] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:19:13 (1786940353) [ 855.396157] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 855.955718] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:19:14 (1786940354) [ 855.991028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.001782] LNet: 112799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.005471] LNetError: 112799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.013746] LNet: Removed LNI 192.168.203.54@tcp [ 856.023458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.046516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.056230] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 856.312405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 856.329328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 858.458793] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:19:17 (1786940357) [ 858.495491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.504116] LNet: 113508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.506588] LNetError: 113508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.513557] LNet: Removed LNI 192.168.203.54@tcp [ 858.525517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.547659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 858.559621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 858.801216] Key type .llcrypt unregistered [ 858.802253] Key type ._llcrypt unregistered [ 862.408285] Key type ._llcrypt registered [ 862.409230] Key type .llcrypt registered [ 862.879341] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:19:21 (1786940361) [ 863.003346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 863.008597] alg: No test for adler32 (adler32-zlib) [ 863.810142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.830568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.842838] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 864.109301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 864.133874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 866.397517] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:19:25 (1786940365) [ 866.435273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.446490] LNet: 115066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.449612] LNetError: 115066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.457928] LNet: Removed LNI 192.168.203.54@tcp [ 866.467698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.489624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 866.504658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 868.513159] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:19:27 (1786940367) [ 868.548789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.560447] LNet: 115644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.574170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.608773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 868.748223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 868.760827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 868.780462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 868.884264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 868.895150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 868.919110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.931431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.952925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 869.068439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 869.081124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 869.102312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 869.221551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 869.237311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 871.214950] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:19:30 (1786940370) [ 871.254200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.263767] LNet: 116278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.266195] LNet: 116278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 871.276804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.301131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.309896] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 871.550609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 871.557660] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 871.844466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 871.863125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 871.935528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.944405] LNetError: 116560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.947126] LNet: Removed LNI 192.168.203.54@tcp [ 871.954326] LNet: Removed LNI 192.168.203.54@tcp2 [ 871.962177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.984663] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 871.988419] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 871.990899] LNet: 116566:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 872.145488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 872.212127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.247133] LNet: Removed LNI 192.168.203.54@tcp [ 873.248364] LNetError: 116603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.250392] LNetError: 116603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 873.258300] LNet: Removed LNI 192.168.203.54@tcp2 [ 873.265489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.285571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.294874] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 873.530287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 873.538028] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 873.817266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 873.834186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 873.859256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 873.943319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.952552] LNet: 116887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.954952] LNet: 116887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 874.975161] LNet: Removed LNI 192.168.203.54@tcp [ 876.005342] LNet: Removed LNI 192.168.203.54@tcp2 [ 876.016940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.044076] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 876.047632] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 876.050694] LNet: 116894:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.204701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 876.273651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.282532] LNetError: 116931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.285452] LNetError: 116931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 876.288385] LNet: Removed LNI 192.168.203.54@tcp [ 877.349530] LNet: Removed LNI 192.168.203.54@tcp2 [ 877.362210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.390385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.405332] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 877.696168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 877.702983] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 877.960656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 877.981307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 878.052367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.059318] LNet: 117210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.061477] LNet: 117210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 879.071161] LNet: Removed LNI 192.168.203.54@tcp [ 879.078331] LNet: Removed LNI 192.168.203.54@tcp2 [ 879.088319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.113610] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 879.115780] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 879.118187] LNet: 117216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 879.269240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 879.341552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.415142] LNet: Removed LNI 192.168.203.54@tcp [ 880.416336] LNetError: 117253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.418328] LNetError: 117253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 880.428375] LNet: Removed LNI 192.168.203.54@tcp2 [ 880.437576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.456522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.466092] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 880.709848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 880.717713] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 881.013744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 881.030827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 881.048902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 881.127672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.143178] LNet: Removed LNI 192.168.203.54@tcp [ 883.173269] LNet: Removed LNI 192.168.203.54@tcp2 [ 883.181176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.202511] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 883.205387] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 883.207834] LNet: 117544:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 883.362590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 883.428874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.435971] LNet: Removed LNI 192.168.203.54@tcp [ 884.453290] LNet: Removed LNI 192.168.203.54@tcp2 [ 884.460403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.480771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.492370] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 884.742988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 884.751216] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 885.045813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 885.061136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 885.084195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 885.206087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.239186] LNet: Removed LNI 192.168.203.54@tcp [ 886.249315] LNet: Removed LNI 192.168.203.54@tcp2 [ 886.257985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.281573] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 886.285090] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 886.441672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 886.554759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.562637] LNet: 117935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.565167] LNet: 117935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 887.583136] LNet: Removed LNI 192.168.203.54@tcp [ 887.591306] LNet: Removed LNI 192.168.203.54@tcp2 [ 887.598371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.616618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.624564] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 887.872223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 887.879724] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 888.173825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 888.187476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 888.200145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 888.222422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 888.339356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.375139] LNet: Removed LNI 192.168.203.54@tcp [ 889.376660] LNetError: 118236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.380214] LNetError: 118236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 889.390267] LNet: Removed LNI 192.168.203.54@tcp2 [ 889.398266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.420381] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 889.422850] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 889.425357] LNet: 118243:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 889.429313] LNet: 118243:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 889.596500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 891.790102] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 892.324919] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:19:51 (1786940391) [ 892.356224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.365609] LNet: Removed LNI 192.168.203.54@tcp [ 893.413302] LNet: Removed LNI 192.168.203.54@tcp2 [ 893.420950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.512047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.520833] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 893.759417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 893.768826] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.072692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.079958] LNet: Removed LNI 10.1.2.3@tcp [ 894.084269] LNet: Removed LNI 192.168.203.54@tcp [ 894.092223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.114756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 894.124873] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 894.126648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.263930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 894.272524] LNet: Removed LNI 192.168.203.54@tcp [ 894.284354] LNet: Removed LNI 10.1.2.3@tcp [ 896.320369] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:19:55 (1786940395) [ 896.356326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.375939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.402471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.413568] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 896.674895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 896.693412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.711426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.725092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.739398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.753113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.766405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.778125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.789447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.801792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.814742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.54@tcp [ 896.862505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 898.949215] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:19:58 (1786940398) [ 898.986261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.009630] LNet: Removed LNI 192.168.203.54@tcp [ 900.018112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.050253] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.087693] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 900.091719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 900.093219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.226973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.235053] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 902.469448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 902.476712] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 903.126509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 903.140781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 903.161257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.54@tcp -m PUT -e local_timeout -r 1 [ 903.183517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 903.204498] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 903.206391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 903.292554] LNetError: 120678:0:(lib-move.c:4801:lnet_parse()) 192.168.203.54@tcp, src 192.168.203.54@tcp: Dropping PUT (error -108 looking up sender) [ 904.351334] LNet: 121141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.353592] LNet: 121141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 904.356615] LNet: Removed LNI 10.1.2.3@tcp [ 905.381381] LNet: Removed LNI 192.168.203.54@tcp [ 905.523629] Key type .llcrypt unregistered [ 905.525425] Key type ._llcrypt unregistered [ 909.149519] Key type ._llcrypt registered [ 909.151395] Key type .llcrypt registered [ 909.590218] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:20:08 (1786940408) [ 909.704641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.710529] alg: No test for adler32 (adler32-zlib) [ 910.514850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.650689] LNet: 121992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.886310] Key type .llcrypt unregistered [ 910.887395] Key type ._llcrypt unregistered [ 912.849357] Key type ._llcrypt registered [ 912.850479] Key type .llcrypt registered [ 912.913070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.918162] alg: No test for adler32 (adler32-zlib) [ 913.717581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 913.727525] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 916.441928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 916.495684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 916.679754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.154@tcp [ 916.759876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 916.771232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 916.783502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 916.794403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 916.805691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 916.816338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 916.827275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 916.832109] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.154@tcp [ 916.842336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 916.849126] LNet: 122444:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 916.856337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 916.869031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 916.880409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 916.892206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 916.904852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 916.918080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.140948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 917.222558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.234407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 917.246981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.257434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.269976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.282492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.293368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.305830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.317213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.329296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.340788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.350790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.361574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.372705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.580184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.154@tcp [ 917.656217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.667617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 917.679907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.692863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.703947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.714791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.725683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.737502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.744159] LNet: 122678:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 917.746278] LNet: 122678:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 917.752782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.763630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.774689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.785482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.800487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.812524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.028158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 918.109053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 918.118886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 918.130082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 918.140241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 918.150606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 918.161377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 918.171660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 918.182322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 918.195322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 918.210794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 918.224873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 918.237920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 918.254739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.270154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 918.393965] LNet: 122845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.396615] LNetError: 122845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.404307] LNet: Removed LNI 192.168.203.54@tcp [ 918.534604] Key type .llcrypt unregistered [ 918.535716] Key type ._llcrypt unregistered [ 923.768575] Key type ._llcrypt registered [ 923.769731] Key type .llcrypt registered [ 924.197012] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:20:23 (1786940423) [ 924.413497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.419391] alg: No test for adler32 (adler32-zlib) [ 925.221078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.347759] LNet: 123730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.570394] Key type .llcrypt unregistered [ 925.571708] Key type ._llcrypt unregistered [ 927.692497] Key type ._llcrypt registered [ 927.693544] Key type .llcrypt registered [ 927.764423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 927.771986] alg: No test for adler32 (adler32-zlib) [ 928.573189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 928.583395] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 931.385642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.154@tcp [ 934.157276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.168390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 934.176711] LNet: Added LNI 192.168.203.54@tcp1 [8/256/0/180] [ 934.466281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 934.655406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.154@tcp [ 934.748684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.751087] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.154@tcp [ 934.760197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 934.772208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.785338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.795675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.806787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.818534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 934.830290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 934.837463] LNet: 124383:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.844533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 934.854457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 934.866404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 934.878957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 934.889944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.902198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.106533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 935.186212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 935.196176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 935.207150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 935.217119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 935.226855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.236916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.246906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.257521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.268986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.281852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.293845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.305325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.316555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.329368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.545547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.154@tcp [ 935.621422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 935.632356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 935.643928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 935.655933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 935.668491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.678633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.691241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.707851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.716514] LNet: 124617:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 935.719853] LNet: 124617:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 935.728471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.743486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.759367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.776185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.792748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.807603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.063962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 936.149752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 936.160796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 936.171978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 936.183586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 936.195973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 936.207544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 936.218548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 936.231503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 936.243026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 936.254858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 936.268836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 936.280070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 936.290590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 936.302429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 938.388248] LNet: 124819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.390450] LNetError: 124819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.393972] LNet: Removed LNI 192.168.203.54@tcp [ 938.400325] LNet: Removed LNI 192.168.203.54@tcp1 [ 938.532551] Key type .llcrypt unregistered [ 938.533940] Key type ._llcrypt unregistered [ 943.872760] Key type ._llcrypt registered [ 943.874568] Key type .llcrypt registered [ 944.318941] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:20:43 (1786940443) [ 944.431151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 944.436035] alg: No test for adler32 (adler32-zlib) [ 945.238819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.258553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.268952] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 945.516343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 945.525629] LNet: Added LNI 192.168.203.54@tcp2 [8/256/0/180] [ 945.856165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.54@tcp [ 945.884292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 946.911161] LNet: Removed LNI 192.168.203.54@tcp [ 948.935355] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:20:47 (1786940447) [ 949.407206] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 949.933519] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:20:49 (1786940449) [ 952.156859] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:20:51 (1786940451) [ 954.406669] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:20:53 (1786940453) [ 956.625821] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:20:55 (1786940455) [ 958.940688] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:20:57 (1786940457) [ 961.234554] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:21:00 (1786940460) [ 963.581346] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:21:02 (1786940462) [ 965.885978] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:21:04 (1786940464) [ 968.097846] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:21:07 (1786940467) [ 970.233725] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:21:09 (1786940469) [ 972.410465] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:21:11 (1786940471) [ 974.665708] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:21:13 (1786940473) [ 976.919938] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:21:15 (1786940475) [ 979.236686] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:21:18 (1786940478) [ 981.418926] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:21:20 (1786940480) [ 983.635146] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:21:22 (1786940482) [ 985.897595] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:21:24 (1786940484) [ 988.148229] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:21:27 (1786940487) [ 988.179294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.189876] LNet: 136489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.192302] LNetError: 136489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.201537] LNet: Removed LNI 192.168.203.54@tcp2 [ 988.211227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.256887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 988.264715] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 988.635852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.655454] LNet: Removed LNI 192.168.203.54@tcp [ 988.665103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.689740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 988.701946] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 990.942791] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:21:29 (1786940489) [ 993.271843] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:21:32 (1786940492) [ 995.518965] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:21:34 (1786940494) [ 997.785931] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:21:36 (1786940496) [ 1000.091526] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:21:39 (1786940499) [ 1002.367127] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:21:41 (1786940501) [ 1004.764343] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:21:43 (1786940503) [ 1007.084468] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:21:46 (1786940506) [ 1009.429557] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:21:48 (1786940508) [ 1011.700036] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:21:50 (1786940510) [ 1013.994074] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:21:53 (1786940513) [ 1016.204699] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:21:55 (1786940515) [ 1018.367157] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 908 sec =========== 00:21:57 (1786940517) [ 1018.456587] LNet: 144094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.459671] LNet: 144094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1018.461985] LNetError: 144094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.464093] LNetError: 144094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1018.473559] LNet: Removed LNI 192.168.203.54@tcp [ 1018.685566] Key type .llcrypt unregistered [ 1018.686784] Key type ._llcrypt unregistered [ 1022.601318] Key type ._llcrypt registered [ 1022.611170] Key type .llcrypt registered [ 1022.740672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.746862] alg: No test for adler32 (adler32-zlib) [ 1023.605757] Lustre: Lustre: Build Version: 2.17.57_2_gd3fb4ca [ 1023.696832] LNet: Added LNI 192.168.203.54@tcp [8/256/0/180] [ 1025.279187] Key type lgssc registered [ 1025.650608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1048.256775] Lustre: Mounted lustre-client - version 2.17.57_2_gd3fb4ca [ 1050.606721] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1052.698132] Lustre: Unmounted lustre-client [ 1077.839969] Key type lgssc unregistered [ 1077.998544] LNet: 147042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.003846] LNetError: 147042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.016324] LNet: Removed LNI 192.168.203.54@tcp [ 1078.356127] Key type .llcrypt unregistered [ 1078.357860] Key type ._llcrypt unregistered