[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424505821 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003242] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007772] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008026] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012041] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.021813] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026275] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031145] *** VALIDATE tmpfs *** [ 0.033214] *** VALIDATE proc *** [ 0.034254] *** VALIDATE cgroup *** [ 0.035024] *** VALIDATE cgroup2 *** [ 0.036277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038078] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040045] Spectre V2 : User space: Vulnerable [ 0.041016] Speculative Store Bypass: Vulnerable [ 0.044684] debug: unmapping init [mem 0xffffffff91a59000-0xffffffff91a60fff] [ 0.047210] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048842] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049032] ... version: 2 [ 0.050021] ... bit width: 48 [ 0.051022] ... generic registers: 4 [ 0.052022] ... value mask: 0000ffffffffffff [ 0.053022] ... max period: 00007fffffffffff [ 0.054028] ... fixed-purpose events: 3 [ 0.055023] ... event mask: 000000070000000f [ 0.057342] rcu: Hierarchical SRCU implementation. [ 0.059842] smp: Bringing up secondary CPUs ... [ 0.060741] x86: Booting SMP configuration: [ 0.061044] .... node #0, CPUs: #1 #2 #3 [ 0.065020] smp: Brought up 1 node, 4 CPUs [ 0.067028] smpboot: Max logical packages: 1 [ 0.068029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.273863] node 0 deferred pages initialised in 202ms [ 0.279159] devtmpfs: initialized [ 0.280296] x86/mm: Memory block size: 128MB [ 0.282345] gcov: version magic: 0x41383552 [ 0.285120] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.287088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.290247] pinctrl core: initialized pinctrl subsystem [ 0.291165] [ 0.291557] ************************************************************* [ 0.293011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296012] ** ** [ 0.297009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.299015] ** ** [ 0.300010] ** This means that this kernel is built to expose internal ** [ 0.302012] ** IOMMU data structures, which may compromise security on ** [ 0.303010] ** your system. ** [ 0.305015] ** ** [ 0.307011] ** If you see this message and you are not debugging the ** [ 0.308009] ** kernel, report this immediately to your vendor! ** [ 0.310025] ** ** [ 0.311010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.313023] ************************************************************* [ 0.314643] NET: Registered protocol family 16 [ 0.316392] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.318054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.319143] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322139] cpuidle: using governor menu [ 0.323694] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325490] PCI: Using configuration type 1 for base access [ 0.326129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.335063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.336022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.338085] cryptd: max_cpu_qlen set to 1000 [ 0.340443] ACPI: Added _OSI(Module Device) [ 0.342018] ACPI: Added _OSI(Processor Device) [ 0.343018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.344015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349412] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355571] ACPI: Interpreter enabled [ 0.357076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.358017] ACPI: Using IOAPIC for interrupt routing [ 0.360209] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.362408] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.374016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.376078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.379350] acpiphp: Slot [2] registered [ 0.380075] acpiphp: Slot [5] registered [ 0.380949] acpiphp: Slot [6] registered [ 0.382080] acpiphp: Slot [3] registered [ 0.382966] acpiphp: Slot [4] registered [ 0.384158] acpiphp: Slot [7] registered [ 0.384983] acpiphp: Slot [8] registered [ 0.385072] acpiphp: Slot [9] registered [ 0.385949] acpiphp: Slot [10] registered [ 0.387083] acpiphp: Slot [11] registered [ 0.388083] acpiphp: Slot [12] registered [ 0.389115] acpiphp: Slot [13] registered [ 0.390096] acpiphp: Slot [14] registered [ 0.391133] acpiphp: Slot [15] registered [ 0.392170] acpiphp: Slot [16] registered [ 0.394118] acpiphp: Slot [17] registered [ 0.395107] acpiphp: Slot [18] registered [ 0.396109] acpiphp: Slot [19] registered [ 0.397093] acpiphp: Slot [20] registered [ 0.398156] acpiphp: Slot [21] registered [ 0.399184] acpiphp: Slot [22] registered [ 0.400115] acpiphp: Slot [23] registered [ 0.401109] acpiphp: Slot [24] registered [ 0.402106] acpiphp: Slot [25] registered [ 0.404080] acpiphp: Slot [26] registered [ 0.405117] acpiphp: Slot [27] registered [ 0.406165] acpiphp: Slot [28] registered [ 0.407129] acpiphp: Slot [29] registered [ 0.408170] acpiphp: Slot [30] registered [ 0.409090] acpiphp: Slot [31] registered [ 0.410057] PCI host bridge to bus 0000:00 [ 0.411016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.413021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.414042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.416034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.418032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.420033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.421209] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.424668] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.427376] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.434015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.436927] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.439019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.444402] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.446369] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.448049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.449000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.452972] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.462021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.465652] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.470083] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479933] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.486019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.500018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.513555] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.519015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.524000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537920] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.540340] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.541400] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.543308] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.546205] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.552032] iommu: Default domain type: Passthrough [ 0.553400] SCSI subsystem initialized [ 0.555132] ACPI: bus type USB registered [ 0.556110] usbcore: registered new interface driver usbfs [ 0.558200] usbcore: registered new interface driver hub [ 0.561084] usbcore: registered new device driver usb [ 0.562145] pps_core: LinuxPPS API ver. 1 registered [ 0.564012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.567069] PTP clock support registered [ 0.569087] EDAC MC: Ver: 3.0.0 [ 0.571145] PCI: Using ACPI for IRQ routing [ 0.573290] NetLabel: Initializing [ 0.575013] NetLabel: domain hash size = 128 [ 0.577017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.581095] NetLabel: unlabeled traffic allowed by default [ 0.584127] vgaarb: loaded [ 0.585406] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.586026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.590780] clocksource: Switched to clocksource kvm-clock [ 0.691438] VFS: Disk quotas dquot_6.6.0 [ 0.692849] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.694757] *** VALIDATE ramfs *** [ 0.695573] *** VALIDATE hugetlbfs *** [ 0.697520] pnp: PnP ACPI init [ 0.699956] pnp: PnP ACPI: found 6 devices [ 0.719047] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.722479] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.725729] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.728383] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.731445] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.734826] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.738392] NET: Registered protocol family 2 [ 0.741368] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.746855] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.749561] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.753679] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.756153] TCP: Hash tables configured (established 65536 bind 65536) [ 0.758076] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.760684] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.765377] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.767851] NET: Registered protocol family 1 [ 0.770088] RPC: Registered named UNIX socket transport module. [ 0.771723] RPC: Registered udp transport module. [ 0.772982] RPC: Registered tcp transport module. [ 0.774262] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.775980] NET: Registered protocol family 44 [ 0.777269] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778973] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780543] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.782307] PCI: CLS 0 bytes, default 64 [ 0.783660] Unpacking initramfs... [ 2.185336] debug: unmapping init [mem 0xffff9319fcc64000-0xffff9319fffcffff] [ 2.188982] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.191386] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.195058] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.702783] Initialise system trusted keyrings [ 2.705026] Key type blacklist registered [ 2.707868] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.716850] zbud: loaded [ 2.720464] *** VALIDATE nfs *** [ 2.721951] *** VALIDATE nfs4 *** [ 2.723944] pstore: using deflate compression [ 2.727588] Platform Keyring initialized [ 2.842410] NET: Registered protocol family 38 [ 2.843948] Key type asymmetric registered [ 2.845161] Asymmetric key parser 'x509' registered [ 2.846565] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.850175] io scheduler mq-deadline registered [ 2.851885] io scheduler kyber registered [ 2.853261] io scheduler bfq registered [ 2.854850] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.861146] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.863672] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.865744] ACPI: Power Button [PWRF] [ 2.871900] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.878570] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.896463] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.924146] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.955718] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.962340] Non-volatile memory driver v1.3 [ 2.964213] Linux agpgart interface v0.103 [ 3.007206] virtio_blk virtio1: [vda] 145800 512-byte logical blocks (74.6 MB/71.2 MiB) [ 3.011936] vda: detected capacity change from 0 to 74649600 [ 3.028080] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.031174] vdb: detected capacity change from 0 to 1073741824 [ 3.040276] libphy: Fixed MDIO Bus: probed [ 3.046215] usbcore: registered new interface driver usbserial_generic [ 3.049064] usbserial: USB Serial support registered for generic [ 3.051664] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.056299] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.058208] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.060957] mousedev: PS/2 mouse device common for all mice [ 3.064123] rtc_cmos 00:05: RTC can wake from S4 [ 3.066781] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.071691] rtc_cmos 00:05: registered as rtc0 [ 3.074279] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.075067] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.078972] intel_pstate: CPU model not supported [ 3.082282] hid: raw HID events driver (C) Jiri Kosina [ 3.087042] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.089148] usbcore: registered new interface driver usbhid [ 3.094056] usbhid: USB HID core driver [ 3.095958] drop_monitor: Initializing network drop monitor service [ 3.098662] Initializing XFRM netlink socket [ 3.100968] NET: Registered protocol family 10 [ 3.104650] Segment Routing with IPv6 [ 3.106227] NET: Registered protocol family 17 [ 3.108439] mpls_gso: MPLS GSO support [ 3.115991] RAS: Correctable Errors collector initialized. [ 3.119418] AVX version of gcm_enc/dec engaged. [ 3.121567] AES CTR mode by8 optimization enabled [ 3.199959] sched_clock: Marking stable (3199938440, 0)->(4106781084, -906842644) [ 3.204086] registered taskstats version 1 [ 3.206396] Loading compiled-in X.509 certificates [ 3.208361] zswap: loaded using pool lzo/zbud [ 3.231259] Key type big_key registered [ 3.244817] Key type encrypted registered [ 3.246756] ima: No TPM chip found, activating TPM-bypass! [ 3.248361] ima: Allocated hash algorithm: sha1 [ 3.249432] ima: No architecture policies found [ 3.250491] evm: Initialising EVM extended attributes: [ 3.251762] evm: security.selinux [ 3.253075] evm: security.ima [ 3.253979] evm: security.capability [ 3.255357] evm: HMAC attrs: 0x1 [ 3.258310] rtc_cmos 00:05: setting system clock to 2026-07-13 12:45:42 UTC (1783946742) [ 3.264657] debug: unmapping init [mem 0xffffffff92a03000-0xffffffff92bfffff] [ 3.267926] debug: unmapping init [mem 0xffffffff91782000-0xffffffff91a58fff] [ 3.277072] Write protecting the kernel read-only data: 28672k [ 3.280934] debug: unmapping init [mem 0xffffffff8fe03000-0xffffffff8fffffff] [ 3.283880] debug: unmapping init [mem 0xffffffff90714000-0xffffffff907fffff] [ 3.316640] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.325489] systemd[1]: Detected virtualization kvm. [ 3.327405] systemd[1]: Detected architecture x86-64. [ 3.329408] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.354515] systemd[1]: No hostname configured. [ 3.356408] systemd[1]: Set hostname to . [ 3.358723] random: systemd: uninitialized urandom read (16 bytes read) [ 3.361321] systemd[1]: Initializing machine ID from random generator. [ 3.491130] random: systemd: uninitialized urandom read (16 bytes read) [ 3.493283] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.497356] random: systemd: uninitialized urandom read (16 bytes read) [ 3.501323] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.507328] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.071771] device-mapper: uevent: version 1.0.3 [ 4.074154] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.801274] virtio_net virtio0 ens2: renamed from eth0 [ 4.912416] scsi host0: ata_piix [ 4.928274] scsi host1: ata_piix [ 4.932504] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.935529] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.328185] dracut-initqueue[589]: RTNETLINK answers: File exists [ 9.761412] random: crng init done [ 9.763079] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.174793] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.364479] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.617171] SELinux: Disabled at runtime. [ 11.677407] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.685495] systemd[1]: Detected virtualization kvm. [ 11.687668] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.169809] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.173589] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.179629] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.184634] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.188419] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.195297] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.200208] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED[ 12.371866] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [0m] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.681759] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.005910] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.094688] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.158411] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.167974] EDAC sbridge: Ver: 1.1.2 [ 14.300601] Key type dns_resolver registered [ 14.599777] NFS: Registering the id_resolver key type [ 14.601681] Key type id_resolver registered [ 14.603073] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-client login: [ 53.093075] libcfs: loading out-of-tree module taints kernel. [ 53.467502] Key type ._llcrypt registered [ 53.475426] Key type .llcrypt registered [ 54.009756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 54.035981] alg: No test for adler32 (adler32-zlib) [ 55.563887] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 56.573843] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 58.335177] Key type lgssc registered [ 60.340471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 105.854970] hrtimer: interrupt took 9828247 ns [ 201.924503] Lustre: Mounted lustre-client [ 207.768281] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 225.294904] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing check_logdir /tmp/testlogs/ [ 227.807646] Lustre: lustre-OST0000-osc-ffff931a60aab800: disconnect after 23s idle [ 230.752575] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing yml_node [ 235.599802] Lustre: DEBUG MARKER: Client: 2.17.54.162 [ 238.676713] Lustre: DEBUG MARKER: MDS: 2.17.54.162 [ 242.029158] Lustre: DEBUG MARKER: OSS: 2.17.54.162 [ 243.666372] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 13 08:49:41 EDT 2026 [ 261.899728] Lustre: Unmounted lustre-client [ 289.759436] Key type lgssc unregistered [ 290.110440] LNet: 8388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.119814] LNetError: 8388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.145019] LNet: Removed LNI 192.168.203.4@tcp [ 291.378360] Key type .llcrypt unregistered [ 291.380980] Key type ._llcrypt unregistered [ 319.112186] Key type ._llcrypt registered [ 319.117476] Key type .llcrypt registered [ 319.503304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 319.543713] alg: No test for adler32 (adler32-zlib) [ 320.563547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 320.661652] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 320.707586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 326.022929] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 326.182838] LNet: 9266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.199579] LNetError: 9266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 326.232943] LNet: Removed LNI 192.168.203.4@tcp [ 327.168398] Key type .llcrypt unregistered [ 327.170096] Key type ._llcrypt unregistered [ 335.786359] Key type ._llcrypt registered [ 335.791936] Key type .llcrypt registered [ 337.633261] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:51:14 (1783947074) [ 338.278846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 338.311931] alg: No test for adler32 (adler32-zlib) [ 339.322433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.443281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.469833] LNet: 9685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.516926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.614940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.646658] LNet: 9692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 347.232687] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:51:25 (1783947085) [ 347.539702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.560120] LNet: 10274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.603785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.743646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 355.492467] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:51:33 (1783947093) [ 355.638456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.668218] LNet: 10857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.692377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.774873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 363.356894] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:51:40 (1783947100) [ 363.721504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.743962] LNet: 11445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.765620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.853459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 371.884456] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:51:49 (1783947109) [ 372.260808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.302977] LNet: 12029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 372.336950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.395476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 373.018942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 380.602669] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:51:58 (1783947118) [ 380.689428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.717936] LNet: 12612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.743313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.845581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 381.482051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 381.914773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 389.978383] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:52:07 (1783947127) [ 390.210854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.304420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.410368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 398.416897] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:52:16 (1783947136) [ 398.785378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.816218] LNet: 13801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.821403] LNet: 13801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 398.843191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.898196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 398.927524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.350709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 399.396451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 400.004416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 400.040825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.580912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 400.642242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.160319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 401.211385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.767575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 401.828192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.281889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 402.332658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.604977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 402.631176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.076795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 403.113691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.577550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 403.612784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.147815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 404.179602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.736659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 404.780717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.280317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 405.325305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.847307] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:52:30 (1783947150) [ 412.959528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.024351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.180916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 413.245994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 421.030490] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:52:38 (1783947158) [ 421.277765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.347366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.434841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 421.474680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 429.405903] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:52:46 (1783947166) [ 429.520939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.581373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.684971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 429.735846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 438.403181] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:52:55 (1783947175) [ 438.690968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.733412] LNet: 16242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.737870] LNet: 16242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 438.765766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.850339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 438.868634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 447.216821] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:53:04 (1783947184) [ 447.639873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.720459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.855858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 447.917970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 458.337074] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:53:15 (1783947195) [ 458.675505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.737886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.892739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 458.945616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.608615] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:53:25 (1783947205) [ 468.732603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.808879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.982604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.466307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 477.079484] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:53:35 (1783947215) [ 477.339726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.391118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.684104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 478.012307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.111856] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:53:42 (1783947222) [ 485.770794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.828668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.043651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.324907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.604588] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:53:54 (1783947234) [ 496.726966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.830617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.801274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.387631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.192353] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:54:04 (1783947244) [ 507.471678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.491699] LNet: 20487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.499831] LNet: 20487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 507.524778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.639900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.702861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.986427] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:54:12 (1783947252) [ 515.142946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.195605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.336703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.403750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.626283] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:54:20 (1783947260) [ 522.965154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.020586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.089612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 530.436744] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:54:28 (1783947268) [ 530.679314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.713678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.767843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 538.562712] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:54:36 (1783947276) [ 538.651221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.710239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.829227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 539.233657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 547.722826] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:54:45 (1783947285) [ 547.885308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.959848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.097959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 548.138604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 555.547627] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:54:53 (1783947293) [ 555.645598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.705948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.839019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 555.884663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 563.911514] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:55:01 (1783947301) [ 564.089697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.173559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.237745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 564.278533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 571.610843] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:55:09 (1783947309) [ 571.870912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.934900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.036676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 572.084332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 579.150302] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:55:16 (1783947316) [ 579.312780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.374487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.498143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 579.576851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 579.594783] LNetError: 25802:0:(peer.c:2276:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 579.650745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 579.699166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 586.941951] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:55:24 (1783947324) [ 587.011877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.054988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.148175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ceW8p81Rtd [ 587.230212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ceW8p81Rtd [ 587.301294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ceW8p81Rtd [ 594.062602] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:55:31 (1783947331) [ 594.332502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.425620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.509187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 594.555364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 594.623446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 601.372043] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:55:39 (1783947339) [ 601.694483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.734195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.818641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.838838] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 603.522232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 605.123793] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 607.207397] Key type lgssc registered [ 608.755996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 613.606664] LNetError: 27563:0:(lib-move.c:2120:lnet_handle_find_routed_path()) peer 192.168.203.4@tcp99 has no available nets [ 616.870777] Key type lgssc unregistered [ 617.321248] LNetError: 28670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.353557] LNet: Removed LNI 192.168.203.4@tcp [ 618.097834] Key type .llcrypt unregistered [ 618.103200] Key type ._llcrypt unregistered [ 625.667600] Key type ._llcrypt registered [ 625.669967] Key type .llcrypt registered [ 627.296547] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:56:04 (1783947364) [ 627.898384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 627.919221] alg: No test for adler32 (adler32-zlib) [ 628.936312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.061446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 632.853182] LNet: Message forwarding will be enabled in 2s [ 635.935567] LNet: Message forwarding enabled [ 636.410795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 636.437976] LNet: Message forwarding will stop in 2s to 6s [ 640.974100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 642.060674] LNet: Message forwarding disabled [ 643.789984] LNet: Message forwarding will be enabled in 2s [ 647.200152] LNet: Message forwarding enabled [ 651.748791] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:56:29 (1783947389) [ 652.165925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.198948] LNet: 30080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.107866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.220082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 661.688037] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:56:39 (1783947399) [ 661.977863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.020490] LNet: 30665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.045423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.116611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 662.168767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 662.234310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 662.265547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 662.297690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 662.342556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 662.369925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 662.406632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 662.436862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 662.474391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 662.511488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 662.544685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 662.584633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 662.630030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 662.723887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 662.805784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 662.856089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 662.918418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 662.958451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 662.978720] LNetError: 30708:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 670.936491] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:56:48 (1783947408) [ 671.101114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.153609] LNet: 31274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.179808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.306410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.549263] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:56:56 (1783947416) [ 681.957242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg304-server [ 684.124879] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:57:01 (1783947421) [ 684.560366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.619082] LNet: 32211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.670222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.763392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.832615] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 686.310110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[5-9]@tcp [ 686.344882] LNet: 32344:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 686.416355] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.5@tcp is being used as a gateway but routing feature is not turned on [ 686.445768] LNetError: 32215:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.5@tcp has gone from up to down [ 687.399362] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 687.420553] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 688.437734] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 688.451952] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 690.474106] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.5@tcp is being used as a gateway but routing feature is not turned on [ 690.483183] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 694.044966] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:57:11 (1783947431) [ 694.570285] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.7@tcp is being used as a gateway but routing feature is not turned on [ 694.582696] LNetError: 32215:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 696.192670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg304-server [ 698.118939] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:57:15 (1783947435) [ 698.256613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.284308] LNet: 33276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.301101] LNetError: 33276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.328715] LNet: Removed LNI 192.168.203.4@tcp [ 698.379933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.442602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 698.468699] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 699.222299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[5-10/2]@tcp [ 699.260454] LNet: 33398:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 699.270875] LNet: 33398:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 699.309951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[5-10/2]@tcp [ 699.341743] LNetError: 33285:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.7@tcp has gone from up to down [ 699.355669] LNetError: 33285:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 706.448732] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:57:24 (1783947444) [ 706.920978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.936489] LNet: 33992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.942564] LNetError: 33992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.971049] LNet: Removed LNI 192.168.203.4@tcp [ 707.006656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.172466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 707.286129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.362604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.488809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 707.573773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.619450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.666142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 707.765248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.826954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.941993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.047559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.136279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.205327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 708.334698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.446849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.553415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 708.623248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.692496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.777897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 708.904754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.971662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.060201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 709.185468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.267985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.341549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 709.471499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.525197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.584464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 709.678646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.707892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.775247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 709.847882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.893904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.990223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.802856] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:57:35 (1783947455) [ 719.725477] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg304-server [ 721.926933] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:57:39 (1783947459) [ 724.097476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg304-server [ 726.231543] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:57:43 (1783947463) [ 726.529354] LNet: 35393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.535798] LNet: 35393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 727.284461] Key type .llcrypt unregistered [ 727.287137] Key type ._llcrypt unregistered [ 734.918863] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.047967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.056471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.062077] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.858475] Key type ._llcrypt registered [ 737.860171] Key type .llcrypt registered [ 738.131969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.167871] alg: No test for adler32 (adler32-zlib) [ 739.142831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.247195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.281953] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 740.115575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.142887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 742.536215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 742.549953] LNet: Removed LNI 10.1.2.3@tcp [ 742.700227] LNet: 36114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.715914] LNetError: 36114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.745236] LNet: Removed LNI 192.168.203.4@tcp [ 743.565485] Key type .llcrypt unregistered [ 743.567441] Key type ._llcrypt unregistered [ 757.319410] Key type ._llcrypt registered [ 757.332180] Key type .llcrypt registered [ 759.010190] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:58:16 (1783947496) [ 759.800285] Key type .llcrypt unregistered [ 759.802786] Key type ._llcrypt unregistered [ 766.501116] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.624535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.640279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.651375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 768.351651] Key type ._llcrypt registered [ 768.362885] Key type .llcrypt registered [ 768.657295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.688168] alg: No test for adler32 (adler32-zlib) [ 769.641803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.725592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.759110] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 770.436302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 770.464634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.204576] LNet: 37612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.213928] LNetError: 37612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 773.227520] LNet: Removed LNI 10.1.2.3@tcp [ 773.242472] LNet: Removed LNI 192.168.203.4@tcp [ 774.082672] Key type .llcrypt unregistered [ 774.086904] Key type ._llcrypt unregistered [ 787.240463] Key type ._llcrypt registered [ 787.242736] Key type .llcrypt registered [ 788.894762] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:58:46 (1783947526) [ 789.432384] Key type .llcrypt unregistered [ 789.435499] Key type ._llcrypt unregistered [ 796.996438] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 797.217908] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 797.225481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 797.235285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 799.934924] Key type ._llcrypt registered [ 799.937829] Key type .llcrypt registered [ 800.158981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 800.185642] alg: No test for adler32 (adler32-zlib) [ 801.149608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.198593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 801.219876] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 803.423251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 803.464970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.535744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 805.564869] LNet: Removed LNI 10.1.2.3@tcp [ 805.592549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 805.631941] LNet: Removed LNI 10.1.2.31@tcp [ 805.651342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 805.681905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 806.390879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 806.428651] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 807.478823] LNet: 39525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.484929] LNetError: 39525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 807.490953] LNet: Removed LNI 10.1.2.31@tcp [ 807.501882] LNet: Removed LNI 10.1.2.3@tcp [ 808.141492] Key type .llcrypt unregistered [ 808.151557] Key type ._llcrypt unregistered [ 821.440360] Key type ._llcrypt registered [ 821.443029] Key type .llcrypt registered [ 822.566615] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:59:20 (1783947560) [ 823.196749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.228484] alg: No test for adler32 (adler32-zlib) [ 824.108675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.159316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.177454] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 824.814552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.852978] LNet: 40478:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.862426] LNetError: 40478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.888744] LNet: Removed LNI 192.168.203.4@tcp [ 824.905283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.966835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.4@tcp [ 824.995895] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 825.755873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.4@tcp [ 825.786961] LNet: Removed LNI 192.168.203.4@tcp [ 825.856383] LNet: 40613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.427440] Key type .llcrypt unregistered [ 826.430486] Key type ._llcrypt unregistered [ 837.028699] Key type ._llcrypt registered [ 837.034399] Key type .llcrypt registered [ 838.456671] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:59:36 (1783947576) [ 840.539617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg304-server [ 842.521656] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:59:40 (1783947580) [ 843.330390] Key type .llcrypt unregistered [ 843.332206] Key type ._llcrypt unregistered [ 850.435806] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.629846] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.638806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.650576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.260864] Key type ._llcrypt registered [ 852.262544] Key type .llcrypt registered [ 852.469533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.491890] alg: No test for adler32 (adler32-zlib) [ 853.444621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.533201] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 854.282237] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 856.661977] LNet: 42442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.668273] LNetError: 42442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.674277] LNet: Removed LNI 192.168.203.4@tcp1 [ 856.684817] LNet: Removed LNI 10.1.2.3@tcp2 [ 857.319576] Key type .llcrypt unregistered [ 857.323260] Key type ._llcrypt unregistered [ 869.173976] Key type ._llcrypt registered [ 869.176034] Key type .llcrypt registered [ 870.208757] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:00:08 (1783947608) [ 870.870462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 870.882260] alg: No test for adler32 (adler32-zlib) [ 871.766832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.809812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.837207] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 872.560459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.605738] LNet: 43411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.614810] LNetError: 43411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.634356] LNet: Removed LNI 192.168.203.4@tcp [ 872.660603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.738400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 872.761965] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 879.888629] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:00:17 (1783947617) [ 880.279554] LNet: 44002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.293350] LNetError: 44002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.314314] LNet: Removed LNI 192.168.203.4@tcp [ 880.893511] Key type .llcrypt unregistered [ 880.895900] Key type ._llcrypt unregistered [ 888.042821] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 888.250798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 888.259290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 888.263940] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.888116] Key type ._llcrypt registered [ 890.893128] Key type .llcrypt registered [ 891.102275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.121742] alg: No test for adler32 (adler32-zlib) [ 892.073264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.145668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.173907] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 893.086843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 893.107677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.827975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.850170] LNet: 44745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.857989] LNetError: 44745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.864778] LNet: Removed LNI 10.1.2.3@tcp [ 895.877216] LNet: Removed LNI 192.168.203.4@tcp [ 895.899195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.983476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 896.015319] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 896.021947] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 896.458519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.510748] LNet: 44772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.519673] LNetError: 44772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.527604] LNetError: 44772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 896.534739] LNet: Removed LNI 10.1.2.3@tcp [ 896.544131] LNet: Removed LNI 192.168.203.4@tcp [ 896.566562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.630960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 896.668465] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 896.677081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.236797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 897.266259] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 898.328682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.345709] LNet: 44935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.351800] LNetError: 44935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.356949] LNetError: 44935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.361857] LNet: Removed LNI 10.1.2.3@tcp [ 898.363342] LNet: Removed LNI 192.168.203.4@tcp [ 898.375886] LNet: Removed LNI 192.168.203.4@tcp2 [ 898.393207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.443915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 898.468200] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 898.472991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 898.478044] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 906.226928] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:00:44 (1783947644) [ 906.386448] LNet: 45529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.397383] LNetError: 45529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.408187] LNetError: 45529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 906.429817] LNet: Removed LNI 10.1.2.3@tcp [ 906.433938] LNet: Removed LNI 192.168.203.4@tcp [ 906.479357] LNet: Removed LNI 192.168.203.4@tcp2 [ 907.154774] Key type .llcrypt unregistered [ 907.157744] Key type ._llcrypt unregistered [ 913.696443] Key type ._llcrypt registered [ 913.698753] Key type .llcrypt registered [ 913.896507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.909895] alg: No test for adler32 (adler32-zlib) [ 914.939875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.076717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.094582] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 916.244952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.276649] LNet: 45928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.289662] LNetError: 45928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.322042] LNet: Removed LNI 192.168.203.4@tcp [ 916.357374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.482522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.526260] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 916.909176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.926391] LNet: 45958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.930586] LNetError: 45958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.948627] LNet: Removed LNI 192.168.203.4@tcp [ 916.973814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.072858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.112279] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 917.739203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.821489] LNet: Removed LNI 192.168.203.4@tcp [ 917.868341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.943680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.985689] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 918.255723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.351665] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.357994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.368619] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.687322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.713618] LNet: 46061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.722773] LNet: 46061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 920.743528] LNetError: 46061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.757512] LNetError: 46061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 920.831932] LNet: Removed LNI 192.168.203.4@tcp [ 920.894744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.009426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 921.042655] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 929.037458] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:01:06 (1783947666) [ 929.247272] LNet: 46663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.280270] LNetError: 46663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.357634] LNet: Removed LNI 192.168.203.4@tcp [ 930.269864] Key type .llcrypt unregistered [ 930.275221] Key type ._llcrypt unregistered [ 938.194240] Key type ._llcrypt registered [ 938.198638] Key type .llcrypt registered [ 938.348821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 938.362230] alg: No test for adler32 (adler32-zlib) [ 939.222777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.257918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 939.287720] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 940.429925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.449799] LNet: 47072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.455533] LNetError: 47072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.469772] LNet: Removed LNI 192.168.203.4@tcp [ 940.494868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.553244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 940.571119] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 948.443451] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:01:26 (1783947686) [ 948.668382] LNet: 47664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.678022] LNetError: 47664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.691333] LNet: Removed LNI 192.168.203.4@tcp [ 949.281243] Key type .llcrypt unregistered [ 949.284567] Key type ._llcrypt unregistered [ 954.786754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 954.948252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 954.955808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 954.966459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 957.494131] Key type ._llcrypt registered [ 957.496152] Key type .llcrypt registered [ 957.679840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.700842] alg: No test for adler32 (adler32-zlib) [ 958.611768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.658937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.680950] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 959.445581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 959.462194] LNet: 48114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.470763] LNetError: 48114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.486801] LNet: Removed LNI 192.168.203.4@tcp [ 959.520688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.612125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 959.643103] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 967.428740] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:01:45 (1783947705) [ 967.582989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.616127] LNet: 48707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.633443] LNetError: 48707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.670899] LNet: Removed LNI 192.168.203.4@tcp [ 967.694214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.821606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 974.300669] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:01:51 (1783947711) [ 974.450317] LNet: 49290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.052520] Key type .llcrypt unregistered [ 975.058300] Key type ._llcrypt unregistered [ 981.223067] Key type ._llcrypt registered [ 981.225564] Key type .llcrypt registered [ 981.411446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.430167] alg: No test for adler32 (adler32-zlib) [ 982.360860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.442853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.477581] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 983.448368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.464768] LNet: 49694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.469647] LNetError: 49694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.483312] LNet: Removed LNI 192.168.203.4@tcp [ 983.497147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.548719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 983.576633] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 990.578403] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:02:08 (1783947728) [ 990.664959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.698825] LNet: 50279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.720245] LNetError: 50279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.767552] LNet: Removed LNI 192.168.203.4@tcp [ 990.810114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.935333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 996.988689] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:02:14 (1783947734) [ 998.360652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 999.980520] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:02:17 (1783947737) [ 1001.736555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1003.289965] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:02:21 (1783947741) [ 1004.775291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1006.552044] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:02:24 (1783947744) [ 1008.449164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1010.571796] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:02:28 (1783947748) [ 1012.250876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1014.029806] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:02:31 (1783947751) [ 1014.263904] LNet: 52336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.102547] Key type .llcrypt unregistered [ 1015.105580] Key type ._llcrypt unregistered [ 1022.617592] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1022.737294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1022.758869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1022.779848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1025.292172] Key type ._llcrypt registered [ 1025.296264] Key type .llcrypt registered [ 1025.502552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.524869] alg: No test for adler32 (adler32-zlib) [ 1026.459737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.537831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1026.557361] LNet: Added LNI 192.168.203.4@tcp [8/512/0/180] [ 1026.562276] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1026.565727] LNet: 52662:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1026.724802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1026.760759] LNet: 52681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.775102] LNetError: 52681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1026.783935] LNet: Removed LNI 10.1.2.3@tcp [ 1026.790306] LNetError: 52681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.815820] LNet: Removed LNI 192.168.203.4@tcp [ 1026.835329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.957284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1026.988306] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1027.108434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.132479] LNet: 52706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.148067] LNetError: 52706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.169167] LNet: Removed LNI 192.168.203.4@tcp [ 1027.209306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.261980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1027.280176] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1027.284702] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1027.287428] LNet: 52714:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1027.367929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.388894] LNet: 52732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.394284] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.399250] LNet: Removed LNI 10.1.2.3@tcp [ 1027.401226] LNetError: 52732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.409938] LNet: Removed LNI 192.168.203.4@tcp [ 1027.430326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.488202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1027.508449] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1027.514912] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1027.517271] LNet: 52740:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1027.696224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.721190] LNet: 52766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.728267] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.731615] LNet: Removed LNI 10.1.2.3@tcp [ 1027.735843] LNetError: 52766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.749964] LNet: Removed LNI 192.168.203.4@tcp [ 1027.767122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.826248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1027.844929] LNet: Added LNI 192.168.203.4@tcp [8/256/0/182] [ 1027.851113] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1027.960232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.976697] LNet: 52794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.982919] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1027.989365] LNet: Removed LNI 10.1.2.3@tcp [ 1027.994504] LNetError: 52794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.023213] LNet: Removed LNI 192.168.203.4@tcp [ 1028.038131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.114581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1028.155090] LNet: Added LNI 192.168.203.4@tcp [8/256/0/182] [ 1028.166387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1028.172246] LNet: 52802:0:(api-ni.c:2931:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1028.257555] LNet: Removed LNI 10.1.2.3@tcp [ 1028.274718] LNet: Removed LNI 192.168.203.4@tcp [ 1028.991515] Key type .llcrypt unregistered [ 1028.996621] Key type ._llcrypt unregistered [ 1040.974669] Key type ._llcrypt registered [ 1040.977355] Key type .llcrypt registered [ 1042.398403] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:03:00 (1783947780) [ 1042.742360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1042.759365] alg: No test for adler32 (adler32-zlib) [ 1043.712146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.780977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1043.807889] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1044.637424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.4 [ 1044.659318] LNet: 53778:0:(config.c:1739:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1044.667334] LNet: Removed LNI 192.168.203.4@tcp [ 1050.888224] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:03:08 (1783947788) [ 1050.979757] LNet: 54341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.001267] LNetError: 54341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.021862] LNet: Removed LNI 192.168.203.4@tcp [ 1051.642429] Key type .llcrypt unregistered [ 1051.648828] Key type ._llcrypt unregistered [ 1057.891228] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1058.067296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1060.716544] Key type ._llcrypt registered [ 1060.720547] Key type .llcrypt registered [ 1060.917309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.955202] alg: No test for adler32 (adler32-zlib) [ 1062.216645] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1069.129786] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:03:26 (1783947806) [ 1069.343960] LNet: 55256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.351312] LNetError: 55256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.378477] LNet: Removed LNI 10.1.2.3@tcp [ 1070.092504] Key type .llcrypt unregistered [ 1070.094343] Key type ._llcrypt unregistered [ 1077.340489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.600889] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1077.605458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.616094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.297769] Key type ._llcrypt registered [ 1080.300398] Key type .llcrypt registered [ 1080.513553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.537626] alg: No test for adler32 (adler32-zlib) [ 1081.509793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.488975] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:03:46 (1783947826) [ 1090.077556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1091.973232] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:03:49 (1783947829) [ 1092.058939] LNet: 56438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1092.069140] LNetError: 56438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1092.098882] LNet: Removed LNI 10.1.2.3@tcp [ 1093.041840] Key type .llcrypt unregistered [ 1093.043662] Key type ._llcrypt unregistered [ 1099.696343] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.984815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1099.988935] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1099.993839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1103.043524] Key type ._llcrypt registered [ 1103.046852] Key type .llcrypt registered [ 1103.392140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1103.431203] alg: No test for adler32 (adler32-zlib) [ 1104.448965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.549289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1114.830723] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:04:12 (1783947852) [ 1115.614479] LNet: 57675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.625274] LNetError: 57675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1115.642902] LNet: Removed LNI 10.1.2.3@tcp [ 1116.550708] Key type .llcrypt unregistered [ 1116.556156] Key type ._llcrypt unregistered [ 1123.623887] Key type ._llcrypt registered [ 1123.625867] Key type .llcrypt registered [ 1123.741928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.750290] alg: No test for adler32 (adler32-zlib) [ 1124.598637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.616075] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1132.241375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1133.151646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_interrupt -r 1 [ 1133.184311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_interrupt -r 1 [ 1133.214595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1133.242297] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1133.441758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.473466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.496367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.524782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.569678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.616662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.687379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.732680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.755137] LNet: 58113:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.780812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.826403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.848815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.860063] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1133.878480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.919654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.965621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.002934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.530418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_dropped -r 1 [ 1134.585269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_dropped -r 1 [ 1134.627293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1134.654535] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1134.900288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.943971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.992184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.033884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.077512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.146816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.193687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.243387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.305693] LNet: 58222:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.335508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.388914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.444802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.489261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.527648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.577684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.612555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1136.144258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_aborted -r 1 [ 1136.188482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_aborted -r 1 [ 1136.231696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1136.255276] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1136.261591] LNet: Skipped 1 previous similar message [ 1136.463992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.501087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1136.550240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.592945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.644604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.677243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.712073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.748338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.783444] LNet: 58332:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.819218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.863763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.902877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.959408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.016649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.049749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.082306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1137.808403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_no_route -r 1 [ 1137.859336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_no_route -r 1 [ 1137.914988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1138.211972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.262713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.307603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.338363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.359632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.434722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.457715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.508589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.545994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.567183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.591371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.612588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.640438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.668209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.693679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1139.306496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_timeout -r 1 [ 1139.364540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_timeout -r 1 [ 1139.423231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1139.461857] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1139.468235] LNet: Skipped 1 previous similar message [ 1139.774111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1139.829656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1139.881869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1139.930366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.989461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.047258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.116741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.185544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.227631] LNet: 58550:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.234687] LNet: 58550:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1140.268443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.309754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.343676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.378712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.427273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.463925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1140.506471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1141.010059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_error -r 1 [ 1141.053630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_error -r 1 [ 1141.093471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1141.336556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1141.371512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1141.394532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.436988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.477794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.518310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.546571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.584459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.625744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.672972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.753684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.802996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.848061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.874549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.898080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1142.190403] LNet: 58700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1142.196356] LNetError: 58700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1142.212861] LNet: Removed LNI 192.168.203.4@tcp [ 1142.673108] Key type .llcrypt unregistered [ 1142.693345] Key type ._llcrypt unregistered [ 1161.555308] Key type ._llcrypt registered [ 1161.558281] Key type .llcrypt registered [ 1163.040841] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:05:00 (1783947900) [ 1163.809401] Key type .llcrypt unregistered [ 1163.817880] Key type ._llcrypt unregistered [ 1170.637395] Key type ._llcrypt registered [ 1170.639825] Key type .llcrypt registered [ 1171.103414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1171.169372] alg: No test for adler32 (adler32-zlib) [ 1172.094650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1172.121597] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1179.406613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1190.209405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.263106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1190.299396] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1191.989922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_interrupt -r 1 [ 1192.038371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_interrupt -r 1 [ 1192.068771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_interrupt -r 1 [ 1192.099758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_interrupt -r 1 [ 1192.146156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_interrupt -r 1 [ 1192.176837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_interrupt -r 1 [ 1192.220961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_interrupt -r 1 [ 1192.267292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_interrupt -r 1 [ 1192.306160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1192.350151] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1192.359625] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1192.520988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.556208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.584534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.632976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.666360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.708862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.758709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.797372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.839582] LNet: 60168:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.858253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.901475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.949348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.988730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.056079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.088351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.138655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.628640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_dropped -r 1 [ 1193.680365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_dropped -r 1 [ 1193.750741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_dropped -r 1 [ 1193.794528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_dropped -r 1 [ 1193.834251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_dropped -r 1 [ 1193.866269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_dropped -r 1 [ 1193.908206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_dropped -r 1 [ 1193.956306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_dropped -r 1 [ 1194.001118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1194.022732] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1194.031052] LNet: Skipped 2 previous similar messages [ 1194.251741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.282707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1194.318211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.366872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.407066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.456319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.499405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.543713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.560257] LNet: 60289:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.585499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.631155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.688847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.714317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.744588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.774186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.807146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.411708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_aborted -r 1 [ 1195.446665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_aborted -r 1 [ 1195.478947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_aborted -r 1 [ 1195.519365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_aborted -r 1 [ 1195.559649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_aborted -r 1 [ 1195.601665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_aborted -r 1 [ 1195.640706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_aborted -r 1 [ 1195.689738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_aborted -r 1 [ 1195.738961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1195.757401] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1195.763679] LNet: Skipped 2 previous similar messages [ 1196.043689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.069681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.097874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.122989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.162906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.208290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.248790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.276854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.292090] LNet: 60410:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.309797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.332893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.360718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.384198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.414824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.453347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.491321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.094977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_no_route -r 1 [ 1197.141878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_no_route -r 1 [ 1197.181697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_no_route -r 1 [ 1197.206801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_no_route -r 1 [ 1197.234556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_no_route -r 1 [ 1197.272426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_no_route -r 1 [ 1197.304711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_no_route -r 1 [ 1197.337680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_no_route -r 1 [ 1197.369688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1197.613540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.656141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1197.701917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.733672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.771845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.812684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.849072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.873099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.913066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.954738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.999994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.042858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.073987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.106906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.143922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.646736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_timeout -r 1 [ 1198.687769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_timeout -r 1 [ 1198.735171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_timeout -r 1 [ 1198.773677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_timeout -r 1 [ 1198.803599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_timeout -r 1 [ 1198.834737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_timeout -r 1 [ 1198.859784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_timeout -r 1 [ 1198.917900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_timeout -r 1 [ 1199.001684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1199.028249] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1199.033721] LNet: Skipped 5 previous similar messages [ 1199.266039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.312240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.346121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.385194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.409466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.433949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.469672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.502860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.519786] LNet: 60653:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.527351] LNet: 60653:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1199.553245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.609585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.652601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.703512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.751829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.781703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.816807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1200.328592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_error -r 1 [ 1200.363373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_error -r 1 [ 1200.401819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_error -r 1 [ 1200.450167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_error -r 1 [ 1200.484856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_error -r 1 [ 1200.523115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_error -r 1 [ 1200.568056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_error -r 1 [ 1200.605037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_error -r 1 [ 1200.643591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1200.912716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.953399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.984727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.029547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.071964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.106468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.136040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.178181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.211062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.247745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.296459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.351808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.402869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.458967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.507508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1206.968442] LNet: 60850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.975723] LNetError: 60850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.983550] LNet: Removed LNI 192.168.203.4@tcp [ 1208.040334] LNet: Removed LNI 192.168.203.4@tcp1 [ 1208.347764] Key type .llcrypt unregistered [ 1208.351782] Key type ._llcrypt unregistered [ 1224.537194] Key type ._llcrypt registered [ 1224.540610] Key type .llcrypt registered [ 1225.602975] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:06:03 (1783947963) [ 1226.476372] Key type .llcrypt unregistered [ 1226.479883] Key type ._llcrypt unregistered [ 1233.367896] Key type ._llcrypt registered [ 1233.370746] Key type .llcrypt registered [ 1233.682530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1233.714800] alg: No test for adler32 (adler32-zlib) [ 1234.637806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1234.663696] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1242.272751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1243.089338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_dropped -r 1 [ 1243.139786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_dropped -r 1 [ 1243.184689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1243.213792] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1243.502577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.552242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.603690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.658915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.705684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.741467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.786371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.832980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.854174] LNet: 62098:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.872738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.932663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.980610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.023956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.068056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.113856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.173683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1244.910962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_error -r 1 [ 1244.972198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_error -r 1 [ 1245.008318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1245.033828] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1245.275738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.313564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.369675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.430682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.481552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.540930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.592687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.668308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.698110] LNet: 62208:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.727682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.758469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.800348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.861070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.922209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.983620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.023481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.633764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_timeout -r 1 [ 1246.700421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_timeout -r 1 [ 1246.729317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1246.754700] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1247.028891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.073097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.135339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.192217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.227415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.262762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.295107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.337516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.355291] LNet: 62317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.367620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.399867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.441246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.474268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.509607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.555475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.586696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1247.912643] LNet: 62358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.918525] LNetError: 62358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.930544] LNet: Removed LNI 192.168.203.4@tcp [ 1248.307492] Key type .llcrypt unregistered [ 1248.309856] Key type ._llcrypt unregistered [ 1265.402367] Key type ._llcrypt registered [ 1265.404466] Key type .llcrypt registered [ 1267.064075] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:06:44 (1783948004) [ 1268.439446] Key type .llcrypt unregistered [ 1268.441487] Key type ._llcrypt unregistered [ 1276.380332] Key type ._llcrypt registered [ 1276.383339] Key type .llcrypt registered [ 1276.693656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.728700] alg: No test for adler32 (adler32-zlib) [ 1277.785600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.850058] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1286.930812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1298.787722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.855983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1298.898173] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1300.234354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_dropped -r 1 [ 1300.265130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e remote_dropped -r 1 [ 1300.295317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_dropped -r 1 [ 1300.328534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e remote_dropped -r 1 [ 1300.361915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e remote_dropped -r 1 [ 1300.400708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e remote_dropped -r 1 [ 1300.443509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e remote_dropped -r 1 [ 1300.498540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e remote_dropped -r 1 [ 1300.568901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1300.598038] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1300.609612] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.104@tcp [ 1300.805441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.832533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1300.870948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.898088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.945488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.981464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.032388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.075200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.094489] LNet: 63825:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.112400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.177666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.231590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.296675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.352645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.395390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1301.442686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1301.498453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.218487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_error -r 1 [ 1302.282594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e remote_error -r 1 [ 1302.370517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_error -r 1 [ 1302.402576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e remote_error -r 1 [ 1302.454893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e remote_error -r 1 [ 1302.498473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e remote_error -r 1 [ 1302.545128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e remote_error -r 1 [ 1302.575380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e remote_error -r 1 [ 1302.619843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1302.645531] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1302.656997] LNet: Skipped 2 previous similar messages [ 1302.932094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.961325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.009697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.032272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.053808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.075972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.098366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1303.120511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1303.143418] LNet: 63948:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1303.170773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1303.200160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1303.236965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1303.269081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.300438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.326994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1303.352587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1303.405588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.039130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_timeout -r 1 [ 1304.102201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e remote_timeout -r 1 [ 1304.156709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_timeout -r 1 [ 1304.211628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e remote_timeout -r 1 [ 1304.282130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e remote_timeout -r 1 [ 1304.332202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e remote_timeout -r 1 [ 1304.385496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e remote_timeout -r 1 [ 1304.437345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e remote_timeout -r 1 [ 1304.502261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.104@tcp [ 1304.526853] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1304.797605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.822878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.844557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.877530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.926462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.963927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.000531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.037903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.056134] LNet: 64072:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.084801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.135094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.181130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.244290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.305121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.338908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.374304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1305.408255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.551327] LNet: 64150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.556503] LNetError: 64150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.568967] LNet: Removed LNI 192.168.203.4@tcp [ 1310.577657] LNet: Removed LNI 192.168.203.4@tcp1 [ 1310.983662] Key type .llcrypt unregistered [ 1310.985293] Key type ._llcrypt unregistered [ 1327.650577] Key type ._llcrypt registered [ 1327.655705] Key type .llcrypt registered [ 1329.093408] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:07:46 (1783948066) [ 1330.592662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1332.839157] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:07:50 (1783948070) [ 1334.008516] Key type .llcrypt unregistered [ 1334.015561] Key type ._llcrypt unregistered [ 1340.316958] Key type ._llcrypt registered [ 1340.323958] Key type .llcrypt registered [ 1340.532916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.552957] alg: No test for adler32 (adler32-zlib) [ 1341.469544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1341.529719] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1349.193547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1349.886260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e network_timeout -r 1 [ 1349.940341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e network_timeout -r 1 [ 1349.979436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1350.005681] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1350.268701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.313545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.343080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.372457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.410332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.446921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.481497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.525058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.556520] LNet: 65692:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.581230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.616386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1350.652952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.681827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.709842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1350.748143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1350.752023] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1351.002757] LNet: 65731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.010192] LNetError: 65731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.042535] LNet: Removed LNI 192.168.203.4@tcp [ 1351.554383] Key type .llcrypt unregistered [ 1351.556616] Key type ._llcrypt unregistered [ 1371.582931] Key type ._llcrypt registered [ 1371.586264] Key type .llcrypt registered [ 1371.791885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1371.806429] alg: No test for adler32 (adler32-zlib) [ 1372.732765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1372.766573] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1380.989648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1390.291615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.352797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.384777] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1391.906195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e network_timeout -r 1 [ 1391.934920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e network_timeout -r 1 [ 1391.970330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e network_timeout -r 1 [ 1392.025820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e network_timeout -r 1 [ 1392.066363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e network_timeout -r 1 [ 1392.099620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e network_timeout -r 1 [ 1392.139507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e network_timeout -r 1 [ 1392.182429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e network_timeout -r 1 [ 1392.262897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1392.290538] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1392.578322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1392.620810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1392.667452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1392.719416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1392.768740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1392.804194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1392.843534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1392.895376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1392.925059] LNet: 66623:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1392.951849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1392.990289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1393.029315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1393.070674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1393.117709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1393.179526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1397.781964] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1398.278716] LNet: 66697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1398.297878] LNetError: 66697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1398.309745] LNet: Removed LNI 192.168.203.4@tcp [ 1398.338807] LNet: Removed LNI 192.168.203.4@tcp1 [ 1398.797299] Key type .llcrypt unregistered [ 1398.799852] Key type ._llcrypt unregistered [ 1418.101687] Key type ._llcrypt registered [ 1418.105523] Key type .llcrypt registered [ 1420.210452] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:09:17 (1783948157) [ 1420.819790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.851274] alg: No test for adler32 (adler32-zlib) [ 1421.895354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.979041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.013364] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1423.302631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.330576] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1424.372752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 1424.485635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1424.568566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1424.656454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1424.727359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1424.782971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1424.842789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.4@tcp [ 1424.889022] LNetError: 67797:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.4@tcp to recovery [ 1424.994535] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1424.994808] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1426.015736] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1426.197248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1430.114077] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1430.892775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1430.929535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.986630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.003972] LNet: 67894:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.011753] LNetError: 67894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.031429] LNet: Removed LNI 192.168.203.4@tcp [ 1432.035261] LNetError: 67894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.069614] LNet: Removed LNI 192.168.203.4@tcp1 [ 1432.108109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.185027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.205646] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1433.045728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1433.073354] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1434.122055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 1434.180182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.245545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1434.298562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1434.339552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1434.378661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1434.470269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1434.518355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.4@tcp [ 1434.543460] LNetError: 68166:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.4@tcp to recovery [ 1435.232501] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1438.118622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1441.378916] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1441.386902] LNet: Skipped 1 previous similar message [ 1446.108479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1446.141339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1446.178888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.938887] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:09:50 (1783948190) [ 1453.205640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.248586] LNet: 68882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.258798] LNetError: 68882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.278020] LNet: Removed LNI 192.168.203.4@tcp [ 1453.311891] LNet: Removed LNI 192.168.203.4@tcp1 [ 1453.330947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.389625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.427409] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1454.307232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.337321] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1455.334648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 1455.405271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1455.431217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1455.503377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1455.566911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1455.621971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1455.690874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.4@tcp --health 0 [ 1455.872965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1456.416643] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.4@tcp [ 1456.425428] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1456.440486] LNet: Skipped 1 previous similar message [ 1471.600465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.770662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1471.808389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1471.865728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.4@tcp --health 500 [ 1471.989165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1478.739160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1478.804441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.850703] LNet: 69377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.858142] LNetError: 69377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.864814] LNetError: 69377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1478.872758] LNet: Removed LNI 192.168.203.4@tcp [ 1478.885122] LNet: Removed LNI 192.168.203.4@tcp1 [ 1478.932477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.013725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.073982] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1480.100614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1480.151731] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1481.409914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 1481.465294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.551737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1481.588404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1481.643485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1481.689759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1481.750937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1481.799942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.4@tcp --health 0 [ 1482.081044] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1482.095960] LNet: Skipped 3 previous similar messages [ 1483.257857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.713304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.752762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1492.787907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1500.859549] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:10:38 (1783948238) [ 1501.563522] LNet: 70412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.574862] LNetError: 70412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.579899] LNetError: 70412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1502.623303] LNet: Removed LNI 192.168.203.4@tcp [ 1502.641414] LNet: Removed LNI 192.168.203.4@tcp1 [ 1503.529109] Key type .llcrypt unregistered [ 1503.538852] Key type ._llcrypt unregistered [ 1513.495949] Key type ._llcrypt registered [ 1513.501466] Key type .llcrypt registered [ 1513.887251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1513.914796] alg: No test for adler32 (adler32-zlib) [ 1514.958166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1515.005739] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1525.652848] Lustre: DEBUG MARKER: Initial discovery [ 1525.686592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.104@tcp [ 1528.392209] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1528.585829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1528.674465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.104@tcp [ 1528.692224] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1528.711885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1528.751395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.104@tcp -r 1 -m GET -l 3 [ 1528.798617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.104@tcp -r 1 -m GET -e local_error [ 1528.857236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.104@tcp -r 1 -m PUT -l 6 [ 1529.378925] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1530.408605] Lustre: DEBUG MARKER: Force oleg304-client.virtnet to discover 192.168.203.104@tcp (in background) [ 1530.577474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.104@tcp [ 1532.215468] Lustre: DEBUG MARKER: Force oleg304-server to discover 192.168.203.4@tcp [ 1533.668049] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1535.028305] Lustre: DEBUG MARKER: Wait for 71176 [ 1536.878795] Lustre: DEBUG MARKER: Finished wait on 71176 [ 1536.993843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1537.046052] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1537.065861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1537.361760] LNet: 71625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.376504] LNetError: 71625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.400708] LNet: Removed LNI 192.168.203.4@tcp [ 1537.843532] Key type .llcrypt unregistered [ 1537.850487] Key type ._llcrypt unregistered [ 1555.168717] Key type ._llcrypt registered [ 1555.171889] Key type .llcrypt registered [ 1556.784493] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:11:34 (1783948294) [ 1557.505560] Key type .llcrypt unregistered [ 1557.510559] Key type ._llcrypt unregistered [ 1564.495488] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1564.720128] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1564.731267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1564.744984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1567.588293] Key type ._llcrypt registered [ 1567.590369] Key type .llcrypt registered [ 1567.803716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1567.823794] alg: No test for adler32 (adler32-zlib) [ 1568.811641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.897531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.922849] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1569.784449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1569.812510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1579.217834] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:11:56 (1783948316) [ 1579.364627] LNet: 73693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1579.385027] LNetError: 73693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1579.396987] LNet: Removed LNI 10.1.2.3@tcp [ 1579.413846] LNet: Removed LNI 192.168.203.4@tcp [ 1580.306693] Key type .llcrypt unregistered [ 1580.310110] Key type ._llcrypt unregistered [ 1587.760968] Key type ._llcrypt registered [ 1587.770365] Key type .llcrypt registered [ 1588.048956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1588.064680] alg: No test for adler32 (adler32-zlib) [ 1589.040921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.167071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.226419] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1590.105342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1590.126545] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1591.148426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.4@tcp [ 1592.223372] LNetError: Unexpected error -22 connecting to 192.168.203.4@tcp at host 192.168.203.4:988 [ 1598.349876] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:12:16 (1783948336) [ 1598.471101] LNet: 74825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.479022] LNetError: 74825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1598.488485] LNet: Removed LNI 10.1.2.3@tcp [ 1598.499136] LNet: Removed LNI 192.168.203.4@tcp [ 1599.297479] Key type .llcrypt unregistered [ 1599.298790] Key type ._llcrypt unregistered [ 1606.777964] Key type ._llcrypt registered [ 1606.781819] Key type .llcrypt registered [ 1607.072542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1607.089231] alg: No test for adler32 (adler32-zlib) [ 1608.037645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.114574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1608.140920] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1609.119117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1609.141688] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 1610.254701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.4@tcp1 --nid 192.168.203.4@tcp2 [ 1615.089697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp1 192.168.203.4@tcp2 [ 1615.117355] LNetError: 75667:0:(lib-move.c:2063:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.4@tcp1 to 192.168.203.4@tcp2 - Route table may be misconfigured [ 1615.139128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp2 192.168.203.4@tcp1 [ 1622.499398] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:12:40 (1783948360) [ 1622.830791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.858810] LNet: 76241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.866678] LNetError: 76241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.880932] LNet: Removed LNI 192.168.203.4@tcp1 [ 1623.914604] LNet: Removed LNI 192.168.203.4@tcp2 [ 1623.939081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.041257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.100460] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1625.065441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1625.084617] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1626.225964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 1626.260907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.4@tcp [ 1626.294827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp1 192.168.203.4@tcp1 [ 1626.331901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.4@tcp -d 192.168.203.4@tcp -e network_timeout [ 1626.363599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -e network_timeout [ 1626.390691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -e network_timeout [ 1626.425865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -e network_timeout [ 1626.457958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1626.488614] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1626.519719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1626.644929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1626.770970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1627.104861] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1633.644453] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:12:51 (1783948371) [ 1633.764772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.795406] LNet: 77096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.804790] LNetError: 77096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.810236] LNetError: 77096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1633.821414] LNet: Removed LNI 192.168.203.4@tcp [ 1633.848245] LNet: Removed LNI 192.168.203.4@tcp1 [ 1633.890628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.981044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1635.009758] Key type .llcrypt unregistered [ 1635.013468] Key type ._llcrypt unregistered [ 1649.071167] Key type ._llcrypt registered [ 1649.073128] Key type .llcrypt registered [ 1650.481487] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:13:08 (1783948388) [ 1650.944858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.968773] alg: No test for adler32 (adler32-zlib) [ 1651.966285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.712490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1655.690711] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:13:13 (1783948393) [ 1655.772945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.809250] LNet: 78230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.843060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.926889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.972598] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1656.850274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.886623] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1657.961933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1658.015949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp1 [ 1658.046991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp1 [ 1664.599590] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:13:22 (1783948402) [ 1666.732696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg304-server [ 1669.077356] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:13:26 (1783948406) [ 1670.990945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg304-server [ 1672.909723] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:13:30 (1783948410) [ 1675.367569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg304-server [ 1677.309394] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:13:34 (1783948414) [ 1679.397947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg304-server [ 1681.400990] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:13:39 (1783948419) [ 1683.282630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg304-server [ 1685.505808] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:13:42 (1783948422) [ 1688.172196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg304-server [ 1690.174703] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:13:47 (1783948427) [ 1691.849590] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg304-server [ 1694.028986] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:13:51 (1783948431) [ 1696.193489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg304-server [ 1698.008324] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:13:55 (1783948435) [ 1700.278778] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg304-server [ 1702.387396] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:14:00 (1783948440) [ 1702.489373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.522448] LNet: 82110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.527857] LNetError: 82110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.541356] LNet: Removed LNI 192.168.203.4@tcp [ 1703.590488] LNet: Removed LNI 192.168.203.4@tcp1 [ 1703.645863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.751295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.798397] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1704.527349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1704.666550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1704.877668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.927324] LNet: 82263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.946924] LNetError: 82263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.964618] LNetError: 82263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1705.005369] LNet: Removed LNI 192.168.203.4@tcp [ 1705.070738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.132307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1705.181229] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1705.914912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1706.156983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.204179] LNetError: 82408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.239378] LNet: Removed LNI 192.168.203.4@tcp [ 1707.286670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.384650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.424051] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1708.317481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1708.482260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1708.767362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.795224] LNet: 82556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.802606] LNet: 82556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1708.817855] LNetError: 82556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.843419] LNet: Removed LNI 192.168.203.4@tcp [ 1708.889082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.963901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1709.001213] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1710.064930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1711.439810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.553012] LNet: Removed LNI 192.168.203.4@tcp [ 1712.603464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.726839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.771179] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1713.639805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1713.711506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1713.932686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.960243] LNet: 82857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.965539] LNet: 82857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1713.981523] LNetError: 82857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.990602] LNetError: 82857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1714.019540] LNet: Removed LNI 192.168.203.4@tcp [ 1714.051113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.134209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1714.174499] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1715.020215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1715.335339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.391610] LNet: Removed LNI 192.168.203.4@tcp [ 1716.449870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.548395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.585756] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1717.442419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1717.514431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1717.754919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.821866] LNet: Removed LNI 192.168.203.4@tcp [ 1718.877125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.961664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1718.978091] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1719.937346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1720.204698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.264772] LNet: Removed LNI 192.168.203.4@tcp [ 1721.312289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.409899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1721.459988] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1722.386665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1722.483607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1722.848599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.872624] LNet: 83443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.877143] LNet: 83443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1722.887962] LNetError: 83443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.894042] LNetError: 83443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1722.909713] LNet: Removed LNI 192.168.203.4@tcp [ 1722.975989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.056291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1723.096214] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1724.165602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1725.635656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.697127] LNet: Removed LNI 192.168.203.4@tcp [ 1726.752653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.864804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1726.902082] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1727.583571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1727.677595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1728.000864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.067839] LNet: Removed LNI 192.168.203.4@tcp [ 1729.090805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.170912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1729.219187] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1730.007760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 1730.295640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.372233] LNet: Removed LNI 192.168.203.4@tcp [ 1731.428700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.505967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.559237] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1732.349972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1732.457789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.536489] LNet: Removed LNI 192.168.203.4@tcp [ 1732.562900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.642098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.663264] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1733.706904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1733.837760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.885932] LNet: Removed LNI 192.168.203.4@tcp [ 1733.915428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.996839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1734.133619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.177506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.252154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1734.366875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.407461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.482489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1734.592659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.636960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.713324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1734.747317] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1735.573433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.609283] LNet: Removed LNI 192.168.203.4@tcp [ 1735.631864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.731342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1743.379187] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:14:41 (1783948481) [ 1743.653652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.676775] LNet: 84910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.685231] LNet: 84910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1743.721847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.788975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1743.833900] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/180] [ 1745.010396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.029974] LNetError: 85049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.037321] LNetError: 85049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1745.054794] LNet: Removed LNI 192.168.203.4@tcp231 [ 1745.069290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.174584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1745.214527] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/180] [ 1745.225525] LNet: 85057:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.660923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1745.694910] LNet: Removed LNI 192.168.203.4@tcp231 [ 1745.708482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1745.750925] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/180] [ 1747.205988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1747.259417] LNet: Removed LNI 192.168.203.4@tcp231 [ 1747.278531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.400220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1747.443554] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/0] [ 1747.450780] LNet: 85209:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1748.182358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1748.225448] LNet: Removed LNI 192.168.203.4@tcp231 [ 1748.246542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1748.298030] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/0] [ 1749.884763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.941492] LNet: Removed LNI 192.168.203.4@tcp231 [ 1749.995493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.129238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1750.189991] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/60] [ 1750.198509] LNet: 85361:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.672748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1750.734842] LNet: Removed LNI 192.168.203.4@tcp231 [ 1750.749687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1750.779511] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/60] [ 1752.174568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.201945] LNet: Removed LNI 192.168.203.4@tcp231 [ 1752.235610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.319823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1752.359684] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/180] [ 1752.373490] LNet: 85512:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.960168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1753.002677] LNet: Removed LNI 192.168.203.4@tcp231 [ 1753.053554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1753.101420] LNet: Added LNI 192.168.203.4@tcp231 [8/256/0/180] [ 1762.614351] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:15:00 (1783948500) [ 1763.034436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.079143] LNet: Removed LNI 192.168.203.4@tcp231 [ 1763.112535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.252080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1763.312780] LNet: Added LNI 192.168.203.4@tcp232 [8/256/0/180] [ 1764.315520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1764.357754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1764.390924] LNet: Removed LNI 192.168.203.4@tcp232 [ 1764.419339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1764.450680] LNet: Added LNI 192.168.203.4@tcp232 [8/256/0/180] [ 1765.250618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1765.287336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1765.320308] LNet: Removed LNI 192.168.203.4@tcp232 [ 1765.347629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1765.385460] LNet: Added LNI 192.168.203.4@tcp232 [8/256/0/180] [ 1766.234842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1766.325333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1766.357184] LNet: Removed LNI 192.168.203.4@tcp232 [ 1766.372452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1766.409882] LNet: Added LNI 192.168.203.4@tcp232 [8/256/0/180] [ 1767.204773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1767.259044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1767.291226] LNet: Removed LNI 192.168.203.4@tcp232 [ 1767.318396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1774.361146] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:15:12 (1783948512) [ 1775.692452] Key type .llcrypt unregistered [ 1775.696620] Key type ._llcrypt unregistered [ 1782.700471] Key type ._llcrypt registered [ 1782.706877] Key type .llcrypt registered [ 1782.913624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1782.933559] alg: No test for adler32 (adler32-zlib) [ 1784.005857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1784.039912] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1792.425607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1801.188337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.242811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1801.264096] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1802.910129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_interrupt -r 2 [ 1802.969827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_interrupt -r 2 [ 1803.020160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_interrupt -r 2 [ 1803.080745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_interrupt -r 2 [ 1803.139820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_interrupt -r 2 [ 1803.220566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_interrupt -r 2 [ 1803.295391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_interrupt -r 2 [ 1803.350288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_interrupt -r 2 [ 1803.406110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1803.426236] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1803.432280] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1803.785695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1803.832973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1803.875899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1803.919870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1803.962401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1804.002397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1804.047721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1804.096511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1804.115930] LNet: 88014:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1804.137763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1804.176944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1804.217804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1804.272974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1804.308032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1804.368463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1805.133867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_dropped -r 2 [ 1805.167465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_dropped -r 2 [ 1805.198995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_dropped -r 2 [ 1805.241674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_dropped -r 2 [ 1805.290975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_dropped -r 2 [ 1805.338847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_dropped -r 2 [ 1805.377536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_dropped -r 2 [ 1805.424724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_dropped -r 2 [ 1805.470820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1805.523531] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1805.830716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1805.860927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.885832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.909667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.940422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.973842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1806.001149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1806.026772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1806.040489] LNet: 88134:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1806.065924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1806.095677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1806.133345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.182445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1806.229796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.265698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1807.052558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_aborted -r 2 [ 1807.097754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_aborted -r 2 [ 1807.138148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_aborted -r 2 [ 1807.190575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_aborted -r 2 [ 1807.276554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_aborted -r 2 [ 1807.358729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_aborted -r 2 [ 1807.427098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_aborted -r 2 [ 1807.474767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_aborted -r 2 [ 1807.519896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1807.553766] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1807.903861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.941992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.981853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1808.019127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1808.056587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1808.088798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1808.122918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.150627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1808.175805] LNet: 88255:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.189157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1808.213602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1808.243832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1808.276289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1808.305367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1808.332499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.959536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_no_route -r 2 [ 1809.026320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_no_route -r 2 [ 1809.108507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_no_route -r 2 [ 1809.164947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_no_route -r 2 [ 1809.221902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_no_route -r 2 [ 1809.306448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_no_route -r 2 [ 1809.374734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_no_route -r 2 [ 1809.429854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_no_route -r 2 [ 1809.474847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1809.793557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1809.858585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1809.920490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1809.960586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1809.991879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1810.057715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1810.146551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1810.246878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1810.299973] LNet: 88375:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1810.331600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1810.364394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1810.411926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.454577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.511462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1810.561764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1811.172456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e local_timeout -r 2 [ 1811.250449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e local_timeout -r 2 [ 1811.319903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e local_timeout -r 2 [ 1811.404885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e local_timeout -r 2 [ 1811.481390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e local_timeout -r 2 [ 1811.533505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e local_timeout -r 2 [ 1811.612877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e local_timeout -r 2 [ 1811.680820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e local_timeout -r 2 [ 1811.739723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1811.765984] LNet: There was an unexpected network error while writing to 192.168.203.104: rc = -22 [ 1811.778245] LNet: Skipped 1 previous similar message [ 1812.151331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.196976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1812.223669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.253459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1812.328361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1812.370078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.424914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.468905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.518616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.552957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.613394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.655183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.705542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.735101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.365974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp -m GET -e remote_dropped -r 2 [ 1813.428731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.104@tcp1 -m GET -e remote_dropped -r 2 [ 1813.530284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp -m GET -e remote_dropped -r 2 [ 1813.612267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp -d 192.168.203.4@tcp1 -m GET -e remote_dropped -r 2 [ 1813.737965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp -m GET -e remote_dropped -r 2 [ 1813.820118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.4@tcp1 -d 192.168.203.104@tcp1 -m GET -e remote_dropped -r 2 [ 1813.889296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp -m GET -e remote_dropped -r 2 [ 1813.985873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.4@tcp1 -d 192.168.203.4@tcp1 -m GET -e remote_dropped -r 2 [ 1814.051406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.4@tcp 192.168.203.104@tcp [ 1814.075716] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.104@tcp [ 1814.314450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1814.339895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1814.363623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1814.390386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1814.449706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1814.499801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1814.542397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1814.572401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1814.613578] LNet: 88615:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1814.623232] LNet: 88615:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1814.672814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1814.730452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1814.841451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1814.923154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1815.024719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1815.089647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.947488] LNet: 88689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.952719] LNetError: 88689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.958615] LNet: Removed LNI 192.168.203.4@tcp [ 1820.968990] LNet: Removed LNI 192.168.203.4@tcp1 [ 1821.405496] Key type .llcrypt unregistered [ 1821.407230] Key type ._llcrypt unregistered [ 1843.270285] Key type ._llcrypt registered [ 1843.272018] Key type .llcrypt registered [ 1844.802878] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:16:22 (1783948582) [ 1845.159927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.173315] alg: No test for adler32 (adler32-zlib) [ 1846.063943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.122159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.150150] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1847.489229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1847.595880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.4@tcp [ 1847.632425] LNetError: 89669:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.4@tcp to recovery [ 1847.805529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.4@tcp [ 1847.920267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1856.481495] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:16:34 (1783948594) [ 1856.616641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.652864] LNet: 90254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.666213] LNetError: 90254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.685408] LNet: Removed LNI 192.168.203.4@tcp [ 1856.722440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.831807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1856.865475] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1857.836030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1857.869250] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 1859.140544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.4@tcp [ 1859.172730] LNetError: 90525:0:(api-ni.c:4418:lnet_ni_set_healthv()) manually adding local NI 192.168.203.4@tcp to recovery [ 1859.871426] LNet: 1 local NIs in recovery (showing 1): 192.168.203.4@tcp [ 1859.872848] LNet: There was an unexpected network error while writing to 192.168.203.4: rc = -22 [ 1860.540853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.4@tcp [ 1870.429409] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:16:48 (1783948608) [ 1872.635521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg304-server [ 1874.586384] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:16:52 (1783948612) [ 1876.847775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg304-server [ 1878.692238] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:16:56 (1783948616) [ 1880.648649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg304-server [ 1882.409054] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:17:00 (1783948620) [ 1882.550176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.590252] LNet: 92151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.601448] LNetError: 92151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.612251] LNet: Removed LNI 192.168.203.4@tcp [ 1882.627029] LNet: Removed LNI 192.168.203.4@tcp1 [ 1882.704749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.775971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1882.806579] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1883.683132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1883.730865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1883.771560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1883.832340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1883.864834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1883.893562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1890.953988] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:17:08 (1783948628) [ 1891.129605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.161063] LNet: 92879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.168872] LNetError: 92879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.176836] LNetError: 92879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1891.210208] LNet: Removed LNI 192.168.203.4@tcp [ 1891.261054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.331360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.368609] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1900.115056] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:17:17 (1783948637) [ 1900.340656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.400093] LNet: 93586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.422820] LNetError: 93586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.471520] LNet: Removed LNI 192.168.203.4@tcp [ 1900.562200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.663540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.709026] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1910.246472] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:17:27 (1783948647) [ 1912.138629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1913.976433] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:17:31 (1783948651) [ 1914.676610] LNet: 94622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.686035] LNetError: 94622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.712772] LNet: Removed LNI 192.168.203.4@tcp [ 1915.694921] Key type .llcrypt unregistered [ 1915.699945] Key type ._llcrypt unregistered [ 1922.949217] Key type ._llcrypt registered [ 1922.957650] Key type .llcrypt registered [ 1923.276676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1923.304697] alg: No test for adler32 (adler32-zlib) [ 1924.279981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1924.314309] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1932.778654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 1938.893906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.104@tcp [ 1939.295753] LNet: 95040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1939.310348] LNetError: 95040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.329638] LNet: Removed LNI 192.168.203.4@tcp [ 1939.856158] Key type .llcrypt unregistered [ 1939.858202] Key type ._llcrypt unregistered [ 1952.176605] Key type ._llcrypt registered [ 1952.179328] Key type .llcrypt registered [ 1953.722978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1955.452829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1957.451686] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:18:15 (1783948695) [ 1959.192196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1961.173961] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:18:18 (1783948698) [ 1963.164602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg304-server [ 1965.223666] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:18:22 (1783948702) [ 1967.461616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg304-server [ 1969.483753] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:18:27 (1783948707) [ 1971.490227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.658736] alg: No test for adler32 (adler32-zlib) [ 1972.872566] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 1973.195678] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 1974.912774] Key type lgssc registered [ 1976.317305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1998.097758] Key type lgssc unregistered [ 1998.470306] LNet: 98214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.474895] LNetError: 98214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.489729] LNet: Removed LNI 192.168.203.4@tcp [ 1999.296199] Key type .llcrypt unregistered [ 1999.300147] Key type ._llcrypt unregistered [ 2019.598619] Key type ._llcrypt registered [ 2019.600660] Key type .llcrypt registered [ 2021.354597] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:19:18 (1783948758) [ 2022.081354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.104871] alg: No test for adler32 (adler32-zlib) [ 2023.112960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.192994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2023.269808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2023.335362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2023.417444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2031.347778] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:19:29 (1783948769) [ 2031.789962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.830737] LNet: 99630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.862222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.922216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2031.952470] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2032.821222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.873522] LNet: 99774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.899126] LNetError: 99774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.941436] LNet: Removed LNI 192.168.203.4@tcp [ 2032.973544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.091486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2033.124263] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2033.912429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.953950] LNet: 99802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.966946] LNetError: 99802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.982977] LNet: Removed LNI 192.168.203.4@tcp [ 2034.036121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.085626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2034.117161] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2041.680277] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:19:39 (1783948779) [ 2042.100564] LNet: 100392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.107988] LNetError: 100392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.133042] LNet: Removed LNI 192.168.203.4@tcp [ 2043.013518] Key type .llcrypt unregistered [ 2043.015979] Key type ._llcrypt unregistered [ 2049.949950] Key type ._llcrypt registered [ 2049.952786] Key type .llcrypt registered [ 2050.142361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.162143] alg: No test for adler32 (adler32-zlib) [ 2051.110071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.177055] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2051.178844] LNet: *** cfs_fail_loc=e003, val=100*** [ 2051.979407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 2052.006160] LNet: *** cfs_fail_loc=e003, val=100*** [ 2052.012283] LNet: Skipped 2 previous similar messages [ 2052.019981] Ping reply received with mapping: [ 2052.023128] found metadata reply NI[192.168.203.4@tcp] [ 2052.026287] 04 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.029563] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.033580] 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.200665] LNet: 100790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.207119] LNetError: 100790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2052.241371] LNet: Removed LNI 192.168.203.4@tcp [ 2053.064381] Key type .llcrypt unregistered [ 2053.067955] Key type ._llcrypt unregistered [ 2065.018331] Key type ._llcrypt registered [ 2065.023396] Key type .llcrypt registered [ 2067.027901] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:20:04 (1783948804) [ 2067.672406] Key type .llcrypt unregistered [ 2067.677468] Key type ._llcrypt unregistered [ 2076.132479] Key type ._llcrypt registered [ 2076.135644] Key type .llcrypt registered [ 2076.342657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2076.366302] alg: No test for adler32 (adler32-zlib) [ 2077.859338] LNetError: 101836:0:(api-ni.c:2795:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2078.603728] Key type .llcrypt unregistered [ 2078.605686] Key type ._llcrypt unregistered [ 2085.642776] Key type ._llcrypt registered [ 2085.653025] Key type .llcrypt registered [ 2087.216791] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:20:24 (1783948824) [ 2088.168608] Key type .llcrypt unregistered [ 2088.171080] Key type ._llcrypt unregistered [ 2094.846812] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2094.851799] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2097.458605] Key type ._llcrypt registered [ 2097.461347] Key type .llcrypt registered [ 2097.770780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2097.799719] alg: No test for adler32 (adler32-zlib) [ 2098.807599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.909971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2109.330247] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:20:47 (1783948847) [ 2109.524406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.542292] LNet: 103614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.562605] LNetError: 103614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2109.577667] LNet: Removed LNI 10.1.2.3@tcp [ 2109.614816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.448949] Key type .llcrypt unregistered [ 2110.451184] Key type ._llcrypt unregistered [ 2117.208779] Key type ._llcrypt registered [ 2117.214301] Key type .llcrypt registered [ 2117.484516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2117.518842] alg: No test for adler32 (adler32-zlib) [ 2118.531227] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2118.605145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 2118.872719] LNet: 103905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.884969] LNetError: 103905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2118.911042] LNet: Removed LNI 192.168.203.4@tcp [ 2119.819702] Key type .llcrypt unregistered [ 2119.825354] Key type ._llcrypt unregistered [ 2132.993117] Key type ._llcrypt registered [ 2132.996375] Key type .llcrypt registered [ 2134.986677] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:21:12 (1783948872) [ 2135.663488] Key type .llcrypt unregistered [ 2135.669734] Key type ._llcrypt unregistered [ 2142.995853] Key type ._llcrypt registered [ 2142.997903] Key type .llcrypt registered [ 2143.315964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.334873] alg: No test for adler32 (adler32-zlib) [ 2150.994420] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:21:28 (1783948888) [ 2151.163318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.222759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.299761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.338225] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2152.202729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.264287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2152.320234] LNet: Removed LNI 192.168.203.4@tcp [ 2152.617312] LNet: 105649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2153.124757] Key type .llcrypt unregistered [ 2153.136992] Key type ._llcrypt unregistered [ 2164.800248] Key type ._llcrypt registered [ 2164.803129] Key type .llcrypt registered [ 2166.215869] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:21:43 (1783948903) [ 2166.645797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.663247] alg: No test for adler32 (adler32-zlib) [ 2167.654312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.719416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2167.756714] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2168.653635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 2168.712449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.4@tcp [ 2168.731281] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2168.736738] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.742805] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 0 [ 2168.751204] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2168.753099] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2168.768336] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.775234] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 0 [ 2168.781545] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2168.784372] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2168.788830] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.793301] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 2218 [ 2168.799688] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2168.804758] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2168.811120] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.815670] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 2218 [ 2168.821580] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2174.823649] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:21:52 (1783948912) [ 2175.414064] LNet: 107189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.420898] LNetError: 107189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.448286] LNet: Removed LNI 192.168.203.4@tcp [ 2176.149388] Key type .llcrypt unregistered [ 2176.153631] Key type ._llcrypt unregistered [ 2182.434582] Key type ._llcrypt registered [ 2182.440903] Key type .llcrypt registered [ 2182.779381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2182.808137] alg: No test for adler32 (adler32-zlib) [ 2183.828713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2183.868356] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2191.778554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 2200.810322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2200.872257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2200.900190] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 2201.866102] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2202.016982] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2202.018875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2202.022064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2204.457709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2204.477194] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2216.989666] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:22:34 (1783948954) [ 2217.233750] LNet: 108694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.237391] LNetError: 108694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.243567] LNet: Removed LNI 192.168.203.4@tcp [ 2218.271364] LNet: Removed LNI 192.168.203.4@tcp1 [ 2218.274126] LNetError: 108694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2218.281436] LNetError: 108694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2218.312873] LNet: Removed LNI 10.1.2.3@tcp99 [ 2219.188204] Key type .llcrypt unregistered [ 2219.193872] Key type ._llcrypt unregistered [ 2226.216288] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2226.224811] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2228.814891] Key type ._llcrypt registered [ 2228.817647] Key type .llcrypt registered [ 2229.006300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.028861] alg: No test for adler32 (adler32-zlib) [ 2230.049696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2230.167704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2230.198360] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2230.802946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2230.838506] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2233.477673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2233.536772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2233.586959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2233.607864] LNetError: 109439:0:(peer.c:2276:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2233.645774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2233.712895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2234.243264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2234.315515] LNet: 109459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.328296] LNetError: 109459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2234.335308] LNet: Removed LNI 10.1.2.3@tcp [ 2235.374485] LNet: Removed LNI 192.168.203.4@tcp [ 2236.033978] Key type .llcrypt unregistered [ 2236.040071] Key type ._llcrypt unregistered [ 2247.586949] Key type ._llcrypt registered [ 2247.590244] Key type .llcrypt registered [ 2248.917753] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:23:06 (1783948986) [ 2249.474409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2249.512973] alg: No test for adler32 (adler32-zlib) [ 2250.434672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.487483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.521505] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2251.084431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.4@tcp [ 2251.156899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg304-client@tcp [ 2257.428220] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:23:15 (1783948995) [ 2257.545599] LNet: 110985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.551875] LNetError: 110985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.571410] LNet: Removed LNI 192.168.203.4@tcp [ 2258.193878] Key type .llcrypt unregistered [ 2258.196415] Key type ._llcrypt unregistered [ 2263.695653] Key type ._llcrypt registered [ 2263.697700] Key type .llcrypt registered [ 2263.905791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2263.927137] alg: No test for adler32 (adler32-zlib) [ 2264.821836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2264.841895] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2264.874354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2270.987691] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:23:28 (1783949008) [ 2272.296764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2274.076591] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:23:31 (1783949011) [ 2275.847832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2277.848984] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:23:35 (1783949015) [ 2277.950984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2277.976984] LNet: 112404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2277.985551] LNetError: 112404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2278.023699] LNet: Removed LNI 192.168.203.4@tcp [ 2278.059639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2278.132514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2278.177702] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2278.946709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2278.993346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2284.574675] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:23:42 (1783949022) [ 2284.868426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.917280] LNet: 113121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.934754] LNetError: 113121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.959872] LNet: Removed LNI 192.168.203.4@tcp [ 2285.004826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.068131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2285.096763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2285.640979] Key type .llcrypt unregistered [ 2285.647137] Key type ._llcrypt unregistered [ 2296.136260] Key type ._llcrypt registered [ 2296.138264] Key type .llcrypt registered [ 2297.367583] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:23:55 (1783949035) [ 2297.750461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.776274] alg: No test for adler32 (adler32-zlib) [ 2298.687123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.745707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.783311] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2299.329186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2299.392390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2305.860714] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:24:03 (1783949043) [ 2306.282649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.311794] LNet: 114685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.321399] LNetError: 114685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.342549] LNet: Removed LNI 192.168.203.4@tcp [ 2306.373495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.432704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2306.463457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2313.076138] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:24:10 (1783949050) [ 2313.357527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2313.390699] LNet: 115265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2313.432141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2313.552559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2313.977321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2314.000932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2314.070798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2314.469779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2314.519646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2314.598425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2314.620704] LNet: 115298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2314.647994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2314.699451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2315.101441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2315.138444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2315.199504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2315.600216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2315.633149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2321.709958] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:24:19 (1783949059) [ 2321.914854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.939581] LNet: 115898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.967722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.024616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.039382] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2322.647940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2322.674673] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2323.352831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2323.414345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2323.581133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.594308] LNetError: 116178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.599436] LNet: Removed LNI 192.168.203.4@tcp [ 2323.614297] LNet: Removed LNI 192.168.203.4@tcp2 [ 2323.640236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.680457] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2323.685841] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2323.690100] LNet: 116184:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2324.117938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2324.317153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.343423] LNet: Removed LNI 192.168.203.4@tcp [ 2325.345261] LNetError: 116221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.352218] LNetError: 116221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2325.368139] LNet: Removed LNI 192.168.203.4@tcp2 [ 2325.395313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.440518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2325.475556] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2326.041756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2326.069193] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2326.909403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2326.977633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2327.060376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2327.241430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.266595] LNet: 116505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.270224] LNet: 116505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2328.288697] LNet: Removed LNI 192.168.203.4@tcp [ 2328.299250] LNetError: 116505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.306201] LNetError: 116505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2328.330266] LNet: Removed LNI 192.168.203.4@tcp2 [ 2328.381149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.445418] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2328.467183] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2328.472378] LNet: 116512:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2328.989598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2329.213422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.232738] LNet: Removed LNI 192.168.203.4@tcp [ 2330.291418] LNet: Removed LNI 192.168.203.4@tcp2 [ 2330.346697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.417949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2330.441555] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2331.173671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2331.195718] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2331.977051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2332.028185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2332.250342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.279484] LNet: Removed LNI 192.168.203.4@tcp [ 2333.284299] LNetError: 116828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.297184] LNetError: 116828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2333.322458] LNet: Removed LNI 192.168.203.4@tcp2 [ 2333.341836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.400643] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2333.406473] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2333.411216] LNet: 116834:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.949368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2334.129505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.199891] LNet: Removed LNI 192.168.203.4@tcp [ 2335.211272] LNet: Removed LNI 192.168.203.4@tcp2 [ 2335.233485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.338412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.378441] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2336.079650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2336.106204] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2337.053590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2337.097765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2337.146780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2337.451259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.493827] LNet: 117156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.505458] LNet: 117156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2338.527858] LNet: Removed LNI 192.168.203.4@tcp [ 2339.557777] LNet: Removed LNI 192.168.203.4@tcp2 [ 2339.605317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.759577] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2339.787594] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2339.795792] LNet: 117162:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.498726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2340.816089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.849657] LNet: Removed LNI 192.168.203.4@tcp [ 2341.867142] LNet: Removed LNI 192.168.203.4@tcp2 [ 2341.893785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.968347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.001725] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2342.696201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.726401] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2343.547927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2343.574936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2343.639160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2343.976716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.995851] LNetError: 117496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.002380] LNetError: 117496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2344.008565] LNet: Removed LNI 192.168.203.4@tcp [ 2344.024932] LNet: Removed LNI 192.168.203.4@tcp2 [ 2344.056229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.112566] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2344.118831] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2344.126412] LNet: 117502:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.598035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2344.935358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.015107] LNet: Removed LNI 192.168.203.4@tcp [ 2346.033210] LNet: Removed LNI 192.168.203.4@tcp2 [ 2346.065632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.149398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2346.186193] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2347.013930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2347.044329] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2347.901503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2347.929486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2347.974608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2348.038600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2348.401064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.471914] LNet: Removed LNI 192.168.203.4@tcp [ 2350.501719] LNet: Removed LNI 192.168.203.4@tcp2 [ 2350.570980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.684381] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2350.694436] LNet: Added LNI 192.168.203.4@tcp2 [8/256/0/180] [ 2351.091689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2358.733451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2361.343172] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:24:58 (1783949098) [ 2361.542172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.592186] LNet: 118621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.602526] LNet: 118621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2361.609778] LNetError: 118621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.622044] LNetError: 118621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2361.630253] LNet: Removed LNI 192.168.203.4@tcp [ 2362.674509] LNet: Removed LNI 192.168.203.4@tcp2 [ 2362.710094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.195427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2363.217305] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2364.198210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2364.232952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2365.352942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.368950] LNet: Removed LNI 10.1.2.3@tcp [ 2365.381088] LNet: Removed LNI 192.168.203.4@tcp [ 2365.396645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.525331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2365.563993] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2365.566745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2366.421624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2366.447610] LNet: Removed LNI 192.168.203.4@tcp [ 2366.461763] LNet: Removed LNI 10.1.2.3@tcp [ 2374.350423] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:25:12 (1783949112) [ 2374.530419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.648394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.788212] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2374.896650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2374.902945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2374.911386] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2377.549347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2377.611582] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2378.561561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2378.607114] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.557074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.4@tcp [ 2381.603671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2381.701336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.4@tcp -m PUT -e local_timeout -r 1 [ 2381.824676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2381.880694] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2381.886726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2382.314649] LNetError: 119563:0:(lib-move.c:4771:lnet_parse()) 192.168.203.4@tcp, src 192.168.203.4@tcp: Dropping PUT (error -108 looking up sender) [ 2383.328528] LNet: Removed LNI 10.1.2.3@tcp [ 2384.359696] LNet: Removed LNI 192.168.203.4@tcp [ 2385.026960] Key type .llcrypt unregistered [ 2385.033468] Key type ._llcrypt unregistered [ 2399.229536] Key type ._llcrypt registered [ 2399.232416] Key type .llcrypt registered [ 2400.521438] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:25:38 (1783949138) [ 2400.872776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2400.893824] alg: No test for adler32 (adler32-zlib) [ 2401.850061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2402.261890] LNet: 120874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2402.847435] Key type .llcrypt unregistered [ 2402.850136] Key type ._llcrypt unregistered [ 2409.481246] Key type ._llcrypt registered [ 2409.487253] Key type .llcrypt registered [ 2409.736125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2409.751119] alg: No test for adler32 (adler32-zlib) [ 2410.636725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2410.677923] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2418.271857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 2418.546389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2419.186318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.104@tcp [ 2419.469884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2419.519420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2419.563725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2419.606394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.648550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.676739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.714576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.749398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.768706] LNet: 121323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.786530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2419.823869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.854964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.888517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.936475] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.104@tcp [ 2419.937701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.966529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2420.568284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2420.827349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.877620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.919468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.952840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.995895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.053815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.104740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.162779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.198627] LNet: 121441:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.215768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.251701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.316976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.379965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.423225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.467323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.116678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.104@tcp [ 2422.391414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.424156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.459516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.490626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.532090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.598613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.648396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.671491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.686339] LNet: 121558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.704657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.742622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.782529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.828666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.863756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.928465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.556535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2423.822379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2423.858523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2423.911511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2423.956599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2423.995572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.038397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.074504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.113835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.159872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.207030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.269401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.320199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.379883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.416717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.697646] LNet: 121721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.705293] LNetError: 121721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.719868] LNet: Removed LNI 192.168.203.4@tcp [ 2425.039826] Key type .llcrypt unregistered [ 2425.042063] Key type ._llcrypt unregistered [ 2441.408874] Key type ._llcrypt registered [ 2441.410827] Key type .llcrypt registered [ 2442.599831] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:26:20 (1783949180) [ 2443.003970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.021702] alg: No test for adler32 (adler32-zlib) [ 2443.951507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.359652] LNet: 122608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.964783] Key type .llcrypt unregistered [ 2444.968545] Key type ._llcrypt unregistered [ 2451.157972] Key type ._llcrypt registered [ 2451.161455] Key type .llcrypt registered [ 2451.385529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2451.401069] alg: No test for adler32 (adler32-zlib) [ 2452.331628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2452.395754] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2459.505923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.104@tcp [ 2468.048519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.107723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2468.166500] LNet: Added LNI 192.168.203.4@tcp1 [8/256/0/180] [ 2469.172680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2469.715723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.104@tcp [ 2469.855565] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.104@tcp [ 2469.906463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2469.929162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2469.954702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2469.978575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2470.006860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2470.038863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2470.061901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2470.094269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2470.108457] LNet: 123260:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2470.136465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2470.188863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2470.225257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2470.254109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2470.303701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2470.345888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2470.926291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2471.134596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2471.155920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2471.181500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2471.208829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2471.233401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2471.267642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2471.308593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2471.333809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2471.362855] LNet: 123377:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2471.380146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2471.420729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2471.461690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2471.515483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2471.556520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2471.594433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2472.160690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.104@tcp [ 2472.358659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2472.389899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2472.426473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2472.452778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2472.480685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2472.507253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2472.529543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2472.562414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2472.578449] LNet: 123494:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2472.596542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2472.634984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2472.693562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2472.720147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2472.752916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2472.791384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2473.401152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2473.673785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2473.703930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2473.741569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2473.770250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2473.797603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2473.820549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2473.844387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2473.871219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2473.902489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2473.935148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2473.972505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2474.002277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2474.033406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2474.063615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2478.979182] LNet: 123692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.990301] LNetError: 123692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2479.005765] LNet: Removed LNI 192.168.203.4@tcp [ 2479.031104] LNet: Removed LNI 192.168.203.4@tcp1 [ 2479.464053] Key type .llcrypt unregistered [ 2479.468141] Key type ._llcrypt unregistered [ 2495.530220] Key type ._llcrypt registered [ 2495.532323] Key type .llcrypt registered [ 2497.235987] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:27:14 (1783949234) [ 2498.860298] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2500.750657] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:27:18 (1783949238) [ 2508.222114] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:27:26 (1783949246) [ 2515.208084] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:27:33 (1783949253) [ 2522.616884] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:27:40 (1783949260) [ 2530.671756] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:27:48 (1783949268) [ 2537.878745] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:27:55 (1783949275) [ 2545.505643] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:28:03 (1783949283) [ 2551.530223] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:28:09 (1783949289) [ 2559.536799] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:28:17 (1783949297) [ 2567.114429] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:28:24 (1783949304) [ 2573.713283] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:28:31 (1783949311) [ 2580.211098] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:28:38 (1783949318) [ 2586.138896] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:28:44 (1783949324) [ 2592.760681] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:28:50 (1783949330) [ 2599.635878] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:28:57 (1783949337) [ 2606.250279] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:29:04 (1783949344) [ 2613.426512] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:29:11 (1783949351) [ 2620.013358] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:29:18 (1783949358) [ 2620.473978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2620.500549] alg: No test for adler32 (adler32-zlib) [ 2621.440895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2621.540454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2621.566041] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2622.729911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2622.765449] LNet: 134690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.771165] LNetError: 134690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.782510] LNet: Removed LNI 192.168.203.4@tcp [ 2622.807735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2622.859795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2622.886227] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2629.990157] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:29:27 (1783949367) [ 2637.260824] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:29:35 (1783949375) [ 2644.695111] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:29:42 (1783949382) [ 2652.033443] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:29:50 (1783949390) [ 2659.244579] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:29:57 (1783949397) [ 2666.968172] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:30:04 (1783949404) [ 2676.370961] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:30:13 (1783949413) [ 2684.833470] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:30:22 (1783949422) [ 2693.692484] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:30:31 (1783949431) [ 2701.583750] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:30:39 (1783949439) [ 2709.215560] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:30:46 (1783949446) [ 2716.426856] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:30:54 (1783949454) [ 2723.750944] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2478 sec ========== 09:31:01 (1783949461) [ 2724.244127] LNet: 142156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2724.249885] LNetError: 142156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.265778] LNet: Removed LNI 192.168.203.4@tcp [ 2724.876964] Key type .llcrypt unregistered [ 2724.881330] Key type ._llcrypt unregistered [ 2739.007368] Key type ._llcrypt registered [ 2739.039146] Key type .llcrypt registered [ 2739.438476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2739.465520] alg: No test for adler32 (adler32-zlib) [ 2740.588415] Lustre: Lustre: Build Version: 2.17.54_162_g6041613 [ 2740.859906] LNet: Added LNI 192.168.203.4@tcp [8/256/0/180] [ 2742.519463] Key type lgssc registered [ 2743.796650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2811.988156] Lustre: Mounted lustre-client [ 2817.243493] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2825.400665] Lustre: Unmounted lustre-client [ 2860.564548] Key type lgssc unregistered [ 2860.867917] LNet: 145087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.878583] LNetError: 145087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.895245] LNet: Removed LNI 192.168.203.4@tcp [ 2861.632152] Key type .llcrypt unregistered [ 2861.637465] Key type ._llcrypt unregistered