[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 423616329 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003336] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010133] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.020832] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024339] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028048] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.031420] *** VALIDATE proc *** [ 0.033198] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036124] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040007] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044278] debug: unmapping init [mem 0xffffffffb4459000-0xffffffffb4460fff] [ 0.046172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049017] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055314] rcu: Hierarchical SRCU implementation. [ 0.057435] smp: Bringing up secondary CPUs ... [ 0.058611] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062383] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.218021] node 0 deferred pages initialised in 151ms [ 0.221267] devtmpfs: initialized [ 0.222219] x86/mm: Memory block size: 128MB [ 0.224818] gcov: version magic: 0x41383552 [ 0.226445] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.227079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.228217] pinctrl core: initialized pinctrl subsystem [ 0.229206] [ 0.229834] ************************************************************* [ 0.230018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.231015] ** ** [ 0.232016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233015] ** ** [ 0.234019] ** This means that this kernel is built to expose internal ** [ 0.235011] ** IOMMU data structures, which may compromise security on ** [ 0.236017] ** your system. ** [ 0.237016] ** ** [ 0.238021] ** If you see this message and you are not debugging the ** [ 0.239014] ** kernel, report this immediately to your vendor! ** [ 0.240014] ** ** [ 0.241015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.242014] ************************************************************* [ 0.243667] NET: Registered protocol family 16 [ 0.244438] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.245062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.246066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.247314] cpuidle: using governor menu [ 0.249391] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.252423] PCI: Using configuration type 1 for base access [ 0.254118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.265110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.266021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.268100] cryptd: max_cpu_qlen set to 1000 [ 0.271682] ACPI: Added _OSI(Module Device) [ 0.274021] ACPI: Added _OSI(Processor Device) [ 0.275030] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.282093] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.289396] ACPI: Interpreter enabled [ 0.290066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.292025] ACPI: Using IOAPIC for interrupt routing [ 0.294129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.297791] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.308970] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.311060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.314028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.318108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.324545] acpiphp: Slot [2] registered [ 0.326129] acpiphp: Slot [5] registered [ 0.328119] acpiphp: Slot [6] registered [ 0.329098] acpiphp: Slot [3] registered [ 0.330122] acpiphp: Slot [4] registered [ 0.332123] acpiphp: Slot [7] registered [ 0.333098] acpiphp: Slot [8] registered [ 0.335121] acpiphp: Slot [9] registered [ 0.337113] acpiphp: Slot [10] registered [ 0.338096] acpiphp: Slot [11] registered [ 0.340101] acpiphp: Slot [12] registered [ 0.341164] acpiphp: Slot [13] registered [ 0.343128] acpiphp: Slot [14] registered [ 0.344094] acpiphp: Slot [15] registered [ 0.346089] acpiphp: Slot [16] registered [ 0.347124] acpiphp: Slot [17] registered [ 0.348113] acpiphp: Slot [18] registered [ 0.350117] acpiphp: Slot [19] registered [ 0.351083] acpiphp: Slot [20] registered [ 0.352110] acpiphp: Slot [21] registered [ 0.354114] acpiphp: Slot [22] registered [ 0.355112] acpiphp: Slot [23] registered [ 0.357086] acpiphp: Slot [24] registered [ 0.358181] acpiphp: Slot [25] registered [ 0.360087] acpiphp: Slot [26] registered [ 0.361153] acpiphp: Slot [27] registered [ 0.363134] acpiphp: Slot [28] registered [ 0.364096] acpiphp: Slot [29] registered [ 0.365086] acpiphp: Slot [30] registered [ 0.367097] acpiphp: Slot [31] registered [ 0.369114] PCI host bridge to bus 0000:00 [ 0.370020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.373033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.376044] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.379051] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.383042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.386061] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.389260] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.392097] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.396584] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.404020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.408065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.412027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.414026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.417026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.420557] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.423982] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430965] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.436014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.447013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.451024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.456903] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.463019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.487021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.498895] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.505016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.511018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.543500] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.546424] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.549433] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.551272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556064] iommu: Default domain type: Passthrough [ 0.557547] SCSI subsystem initialized [ 0.559163] ACPI: bus type USB registered [ 0.560155] usbcore: registered new interface driver usbfs [ 0.563117] usbcore: registered new interface driver hub [ 0.565104] usbcore: registered new device driver usb [ 0.567227] pps_core: LinuxPPS API ver. 1 registered [ 0.569014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.572082] PTP clock support registered [ 0.574257] EDAC MC: Ver: 3.0.0 [ 0.577097] PCI: Using ACPI for IRQ routing [ 0.578829] NetLabel: Initializing [ 0.580009] NetLabel: domain hash size = 128 [ 0.582011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.584103] NetLabel: unlabeled traffic allowed by default [ 0.586188] vgaarb: loaded [ 0.588320] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.590022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.597449] clocksource: Switched to clocksource kvm-clock [ 0.707612] VFS: Disk quotas dquot_6.6.0 [ 0.709502] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.712470] *** VALIDATE ramfs *** [ 0.713874] *** VALIDATE hugetlbfs *** [ 0.716088] pnp: PnP ACPI init [ 0.718500] pnp: PnP ACPI: found 6 devices [ 0.738459] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.742504] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.745271] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.748121] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.750529] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.752772] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.755605] NET: Registered protocol family 2 [ 0.758501] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.763919] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.783425] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.794203] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.797924] TCP: Hash tables configured (established 65536 bind 65536) [ 0.801159] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.803917] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806930] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810195] NET: Registered protocol family 1 [ 0.813639] RPC: Registered named UNIX socket transport module. [ 0.816200] RPC: Registered udp transport module. [ 0.817788] RPC: Registered tcp transport module. [ 0.819715] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.822325] NET: Registered protocol family 44 [ 0.824303] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.826805] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.829064] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.831561] PCI: CLS 0 bytes, default 64 [ 0.833498] Unpacking initramfs... [ 2.306662] debug: unmapping init [mem 0xffff8f713cc64000-0xffff8f713ffcffff] [ 2.310717] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.313789] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.316971] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.866664] Initialise system trusted keyrings [ 2.868614] Key type blacklist registered [ 2.870977] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.882823] zbud: loaded [ 2.886052] *** VALIDATE nfs *** [ 2.887512] *** VALIDATE nfs4 *** [ 2.889229] pstore: using deflate compression [ 2.892893] Platform Keyring initialized [ 3.001677] NET: Registered protocol family 38 [ 3.004433] Key type asymmetric registered [ 3.007130] Asymmetric key parser 'x509' registered [ 3.009776] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.014384] io scheduler mq-deadline registered [ 3.016546] io scheduler kyber registered [ 3.018380] io scheduler bfq registered [ 3.020445] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.023820] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.027406] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.030832] ACPI: Power Button [PWRF] [ 3.036560] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.044921] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.064115] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.092260] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121812] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.126630] Non-volatile memory driver v1.3 [ 3.129212] Linux agpgart interface v0.103 [ 3.164708] virtio_blk virtio1: [vda] 150672 512-byte logical blocks (77.1 MB/73.6 MiB) [ 3.169164] vda: detected capacity change from 0 to 77144064 [ 3.187769] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.190981] vdb: detected capacity change from 0 to 1073741824 [ 3.198283] libphy: Fixed MDIO Bus: probed [ 3.206099] usbcore: registered new interface driver usbserial_generic [ 3.210189] usbserial: USB Serial support registered for generic [ 3.212973] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.217857] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.220022] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.222904] mousedev: PS/2 mouse device common for all mice [ 3.226372] rtc_cmos 00:05: RTC can wake from S4 [ 3.229476] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.231481] rtc_cmos 00:05: registered as rtc0 [ 3.237727] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.239504] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.243163] intel_pstate: CPU model not supported [ 3.249037] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.254909] hid: raw HID events driver (C) Jiri Kosina [ 3.257208] usbcore: registered new interface driver usbhid [ 3.259650] usbhid: USB HID core driver [ 3.261899] drop_monitor: Initializing network drop monitor service [ 3.265578] Initializing XFRM netlink socket [ 3.268562] NET: Registered protocol family 10 [ 3.272624] Segment Routing with IPv6 [ 3.274366] NET: Registered protocol family 17 [ 3.276830] mpls_gso: MPLS GSO support [ 3.281446] RAS: Correctable Errors collector initialized. [ 3.283726] AVX version of gcm_enc/dec engaged. [ 3.285514] AES CTR mode by8 optimization enabled [ 3.372597] sched_clock: Marking stable (3372563906, 0)->(4319643585, -947079679) [ 3.377594] registered taskstats version 1 [ 3.379929] Loading compiled-in X.509 certificates [ 3.384970] zswap: loaded using pool lzo/zbud [ 3.413112] Key type big_key registered [ 3.425941] Key type encrypted registered [ 3.427841] ima: No TPM chip found, activating TPM-bypass! [ 3.430121] ima: Allocated hash algorithm: sha1 [ 3.431545] ima: No architecture policies found [ 3.433533] evm: Initialising EVM extended attributes: [ 3.435518] evm: security.selinux [ 3.436778] evm: security.ima [ 3.437896] evm: security.capability [ 3.439075] evm: HMAC attrs: 0x1 [ 3.441411] rtc_cmos 00:05: setting system clock to 2026-09-30 12:10:02 UTC (1790770202) [ 3.447069] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 3.450579] debug: unmapping init [mem 0xffffffffb4182000-0xffffffffb4458fff] [ 3.461100] Write protecting the kernel read-only data: 28672k [ 3.464594] debug: unmapping init [mem 0xffffffffb2803000-0xffffffffb29fffff] [ 3.467758] debug: unmapping init [mem 0xffffffffb3114000-0xffffffffb31fffff] [ 3.499944] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.509193] systemd[1]: Detected virtualization kvm. [ 3.511410] systemd[1]: Detected architecture x86-64. [ 3.513435] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.542881] systemd[1]: No hostname configured. [ 3.546331] systemd[1]: Set hostname to . [ 3.548824] random: systemd: uninitialized urandom read (16 bytes read) [ 3.551439] systemd[1]: Initializing machine ID from random generator. [ 3.601874] random: ln: uninitialized urandom read (6 bytes read) [ 3.703167] random: systemd: uninitialized urandom read (16 bytes read) [ 3.706089] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.710339] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.714828] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.378413] device-mapper: uevent: version 1.0.3 [ 4.382124] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.139985] virtio_net virtio0 ens2: renamed from eth0 [ 5.190349] scsi host0: ata_piix [ 5.237130] scsi host1: ata_piix [ 5.239121] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.242279] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.031328] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.944732] random: crng init done [ 9.946199] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.481677] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.755869] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.026263] SELinux: Disabled at runtime. [ 12.092844] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.103054] systemd[1]: Detected virtualization kvm. [ 12.104529] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.688483] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.692763] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.699764] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.704432] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.707298] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.714801] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.722173] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Startin[ 12.830390] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.211929] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.531241] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.557120] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.740828] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.768260] EDAC sbridge: Ver: 1.1.2 [ 14.839823] Key type dns_resolver registered [ 15.155065] NFS: Registering the id_resolver key type [ 15.157118] Key type id_resolver registered [ 15.158800] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg653-client login: [ 42.688340] libcfs: loading out-of-tree module taints kernel. [ 42.762108] Key type ._llcrypt registered [ 42.771153] Key type .llcrypt registered [ 43.014404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 43.022093] alg: No test for adler32 (adler32-zlib) [ 44.062342] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 44.431218] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 46.063197] Key type lgssc registered [ 46.771811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 144.072051] hrtimer: interrupt took 5039800 ns [ 164.782353] Lustre: Mounted lustre-client - version 2.17.59_3_gf57cca5 [ 169.436724] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 181.723162] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing check_logdir /tmp/testlogs/ [ 186.064937] Lustre: DEBUG MARKER: oleg653-client.virtnet: executing yml_node [ 189.391313] Lustre: DEBUG MARKER: Client: 2.17.59.3 [ 190.432365] Lustre: lustre-OST0000-osc-ffff8f71863c4800: disconnect after 23s idle [ 191.734804] Lustre: DEBUG MARKER: MDS: 2.17.59.3 [ 193.721811] Lustre: DEBUG MARKER: OSS: 2.17.59.3 [ 195.022282] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 30 08:13:12 EDT 2026 [ 210.057295] Lustre: Unmounted lustre-client [ 230.942995] Key type lgssc unregistered [ 231.125967] LNet: 8381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 231.142230] LNetError: 8381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 232.168992] LNet: Removed LNI 192.168.206.53@tcp [ 232.873133] Key type .llcrypt unregistered [ 232.875063] Key type ._llcrypt unregistered [ 258.512756] Key type ._llcrypt registered [ 258.517555] Key type .llcrypt registered [ 258.766634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 258.782381] alg: No test for adler32 (adler32-zlib) [ 259.744177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 259.814022] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 259.832451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 263.678181] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 263.748319] LNet: 9267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.756784] LNetError: 9267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.781502] LNet: Removed LNI 192.168.206.53@tcp [ 264.376503] Key type .llcrypt unregistered [ 264.380087] Key type ._llcrypt unregistered [ 270.111532] Key type ._llcrypt registered [ 270.115084] Key type .llcrypt registered [ 271.383695] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:14:28 (1790770468) [ 271.918906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 271.938362] alg: No test for adler32 (adler32-zlib) [ 272.855287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.938351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.956120] LNet: 9682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.973836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.032164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 273.055890] LNet: 9689:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 279.374341] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:14:36 (1790770476) [ 279.617580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.657220] LNet: 10266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.685771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.751723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 286.622168] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:14:44 (1790770484) [ 286.843240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.891704] LNet: 10851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.930167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 287.019561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 293.777420] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:14:51 (1790770491) [ 293.929542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.977251] LNet: 11435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.009809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.083857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 300.273275] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:14:57 (1790770497) [ 300.372149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.397480] LNet: 12013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.443681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.588964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 301.273706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 307.183608] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:15:05 (1790770505) [ 307.385704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.434131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.526387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 307.947932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 308.429637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 314.638637] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:15:12 (1790770512) [ 314.723956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.762466] LNet: 13201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.770470] LNet: 13201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 314.807320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.917889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 321.880414] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:15:19 (1790770519) [ 321.964551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.019476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.097184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 322.144911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 322.637386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 322.664953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 323.127378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 323.190069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 323.800364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 323.872598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 324.490734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 324.527703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 324.949922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 324.989776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 325.319852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 325.370520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 325.784139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 325.812604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 326.123109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 326.158557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 326.654878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 326.708556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 327.263316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 327.297877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 327.757579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 327.791782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 328.140845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 328.179467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 334.223564] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:15:32 (1790770532) [ 334.312080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.357781] LNet: 14470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.361276] LNet: 14470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 334.411028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.507436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 334.542785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 340.627832] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:15:38 (1790770538) [ 340.809429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 340.869109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.981851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 341.035680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 347.352396] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:15:45 (1790770545) [ 347.546798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.579421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.647451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 347.687744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 353.600388] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:15:51 (1790770551) [ 353.813528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.864381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.963325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 354.018479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 360.177300] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:15:58 (1790770558) [ 360.310503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.381358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.510102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 360.579962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 366.557577] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:16:04 (1790770564) [ 366.776641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.803217] LNet: 17407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.810796] LNet: 17407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 366.831865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.907355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 366.948165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 372.604237] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:16:10 (1790770570) [ 372.877778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.942981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.032075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 373.616225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 378.935961] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:16:16 (1790770576) [ 379.234872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.278992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.541219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.768369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.477510] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:16:24 (1790770584) [ 387.619363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.706001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.881694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.149482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 394.963623] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:16:32 (1790770592) [ 395.207073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.236343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.734776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 396.069569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 403.130564] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:16:41 (1790770601) [ 403.267352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.295045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.334979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.363847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 410.088299] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:16:47 (1790770607) [ 410.456384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.505653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.610396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.672274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 417.101384] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:16:54 (1790770614) [ 417.288108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.346274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.451693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 423.011557] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:17:00 (1790770620) [ 423.160812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.200835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.263450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 429.507139] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:17:07 (1790770627) [ 429.739584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.790235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.861105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 430.259279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 436.598793] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:17:14 (1790770634) [ 436.961587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.988285] LNet: 23433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.000312] LNet: 23433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 437.041539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.125750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 437.178736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 443.335737] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:17:21 (1790770641) [ 443.607448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.660661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.728493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 443.755600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 449.349863] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:17:27 (1790770647) [ 449.630645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.658864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.749049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 449.785379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 456.464266] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:17:33 (1790770653) [ 456.641471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.721889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.815859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 456.852444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 462.579524] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:17:40 (1790770660) [ 462.832915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.873382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.913915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 462.972971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 462.992438] LNetError: 25790:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 463.030972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 463.057590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 468.730116] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:17:46 (1790770666) [ 468.892571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.918422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.974936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O6tcFCyjVu [ 469.027854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O6tcFCyjVu [ 469.071701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O6tcFCyjVu [ 474.723583] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:17:52 (1790770672) [ 475.049505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.109801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.170701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 475.206323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 475.263508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 481.198523] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:17:59 (1790770679) [ 481.430702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.471502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.532320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 481.552571] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 482.748848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 483.645542] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 485.463238] Key type lgssc registered [ 486.478439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 489.277949] LNetError: 27551:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.206.53@tcp99 has no available nets [ 491.371680] Key type lgssc unregistered [ 491.738700] LNetError: 28664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 491.755428] LNet: Removed LNI 192.168.206.53@tcp [ 492.309116] Key type .llcrypt unregistered [ 492.314330] Key type ._llcrypt unregistered [ 498.377221] Key type ._llcrypt registered [ 498.380602] Key type .llcrypt registered [ 499.563156] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:18:17 (1790770697) [ 500.278268] Key type .llcrypt unregistered [ 500.284228] Key type ._llcrypt unregistered [ 507.055996] Key type ._llcrypt registered [ 507.059880] Key type .llcrypt registered [ 507.208342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 507.226910] alg: No test for adler32 (adler32-zlib) [ 508.236125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.311546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 508.330424] LNet: 29586:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 508.347795] LNet: 29586:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 508.359980] LNet: 29586:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 508.377745] LNet: Message forwarding will be enabled in 2s [ 511.455122] LNet: Message forwarding enabled [ 511.940465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 511.965687] LNet: Message forwarding will stop in 2s to 6s [ 515.563263] LNet: Message forwarding disabled [ 516.482826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 516.505617] LNet: 29684:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 516.511868] LNet: 29684:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 516.516936] LNet: 29684:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 516.528465] LNet: Message forwarding will be enabled in 2s [ 516.667295] LNet: 29697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.112223] Key type .llcrypt unregistered [ 517.114875] Key type ._llcrypt unregistered [ 527.987855] Key type ._llcrypt registered [ 527.990651] Key type .llcrypt registered [ 529.225193] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:18:46 (1790770726) [ 529.806462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 529.822308] alg: No test for adler32 (adler32-zlib) [ 530.671710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.713637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 535.886851] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:18:53 (1790770733) [ 536.235618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.277273] LNet: 31098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.299782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.349392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 536.380871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 536.412255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 536.444210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 536.481162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 536.510649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 536.545064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 536.584538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 536.626279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 536.661517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 536.687754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 536.714507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 536.742426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 536.770849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 536.797665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 536.819867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 536.844228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 536.871527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 536.936213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 536.960707] LNetError: 31141:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 541.939376] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:18:59 (1790770739) [ 542.056950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.075877] LNet: 31708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.112591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.182916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.556352] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:19:05 (1790770745) [ 549.309254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg653-server [ 550.972217] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:19:08 (1790770748) [ 551.201513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.227679] LNet: 32642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.257454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.315459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 551.355491] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 552.096288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[54-58]@tcp [ 552.113881] LNet: 32775:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 552.171360] LNetError: 32646:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.55@tcp is being used as a gateway but routing feature is not turned on [ 552.191864] LNetError: 32646:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.55@tcp has gone from up to down [ 558.400178] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:19:16 (1790770756) [ 560.194318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg653-server [ 562.009803] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:19:19 (1790770759) [ 562.333325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.383582] LNet: 33711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.406663] LNetError: 33711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.430042] LNet: Removed LNI 192.168.206.53@tcp [ 562.453251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.494743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 562.510502] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 563.152940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[54-59/2]@tcp [ 563.181752] LNet: 33828:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 563.190295] LNet: 33828:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 563.212976] LNetError: 33715:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.56@tcp is being used as a gateway but routing feature is not turned on [ 563.228051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[54-59/2]@tcp [ 563.233978] LNetError: 33715:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 563.255498] LNetError: 33715:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.56@tcp has gone from up to down [ 563.265467] LNetError: 33715:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 568.855486] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:19:26 (1790770766) [ 569.171749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.217289] LNet: 34420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.238125] LNetError: 34420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.254466] LNet: Removed LNI 192.168.206.53@tcp [ 569.280326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.387403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 569.452619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.488164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.567964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.620511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.663798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.696722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 569.763743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.790288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.854560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.932128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.971581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.011861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 570.077475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.115379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.188059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.242720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.274502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.321373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 570.389872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.429667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.495649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.561943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.594661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.645550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 570.723864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.767232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.818767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 570.880676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.920315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.969440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 571.017308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.045091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.092283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 576.577902] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:19:34 (1790770774) [ 578.324931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg653-server [ 580.023762] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:19:37 (1790770777) [ 581.667669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg653-server [ 583.450811] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:19:41 (1790770781) [ 583.900498] LNet: 35821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 583.905728] LNet: 35821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 584.529553] Key type .llcrypt unregistered [ 584.531396] Key type ._llcrypt unregistered [ 591.991752] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 592.219245] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 592.229179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 592.239366] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 594.841824] Key type ._llcrypt registered [ 594.847825] Key type .llcrypt registered [ 595.032513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.055615] alg: No test for adler32 (adler32-zlib) [ 595.960890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.998896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.021120] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 596.574361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 596.594520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 598.361445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 598.378457] LNet: Removed LNI 10.1.2.3@tcp [ 598.514644] LNet: 36533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.526078] LNetError: 36533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.541305] LNet: Removed LNI 192.168.206.53@tcp [ 599.174598] Key type .llcrypt unregistered [ 599.177740] Key type ._llcrypt unregistered [ 608.778589] Key type ._llcrypt registered [ 608.780651] Key type .llcrypt registered [ 609.815695] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:20:07 (1790770807) [ 610.527958] Key type .llcrypt unregistered [ 610.532055] Key type ._llcrypt unregistered [ 616.526268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 616.653707] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 616.658566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 616.662308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 619.202361] Key type ._llcrypt registered [ 619.205345] Key type .llcrypt registered [ 619.406290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 619.427231] alg: No test for adler32 (adler32-zlib) [ 620.333153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.430734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 620.466027] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 621.022791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 621.051882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 623.144749] LNet: 38023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.152909] LNetError: 38023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 623.161064] LNet: Removed LNI 10.1.2.3@tcp [ 623.186088] LNet: Removed LNI 192.168.206.53@tcp [ 623.774952] Key type .llcrypt unregistered [ 623.778461] Key type ._llcrypt unregistered [ 635.383659] Key type ._llcrypt registered [ 635.388277] Key type .llcrypt registered [ 636.462393] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:20:34 (1790770834) [ 637.115281] Key type .llcrypt unregistered [ 637.118895] Key type ._llcrypt unregistered [ 643.616481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 643.809824] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 643.816771] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 643.821319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 646.437808] Key type ._llcrypt registered [ 646.440169] Key type .llcrypt registered [ 646.730218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.770991] alg: No test for adler32 (adler32-zlib) [ 647.679414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.770952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 647.813880] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 649.777349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.801033] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.790478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 651.812470] LNet: Removed LNI 10.1.2.3@tcp [ 651.830759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 651.866791] LNet: Removed LNI 10.1.2.31@tcp [ 651.885638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 651.918267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.571679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 652.595579] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 653.565767] LNet: 39939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.570837] LNetError: 39939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 653.578181] LNet: Removed LNI 10.1.2.31@tcp [ 653.593475] LNet: Removed LNI 10.1.2.3@tcp [ 654.186595] Key type .llcrypt unregistered [ 654.188085] Key type ._llcrypt unregistered [ 666.020864] Key type ._llcrypt registered [ 666.023078] Key type .llcrypt registered [ 667.322892] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:21:04 (1790770864) [ 667.645979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.668241] alg: No test for adler32 (adler32-zlib) [ 668.694521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.762952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 668.803158] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 669.603335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.633332] LNet: 40891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.637680] LNetError: 40891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.652851] LNet: Removed LNI 192.168.206.53@tcp [ 669.677733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.793927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.53@tcp [ 669.827766] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 670.469908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.53@tcp [ 670.499884] LNet: Removed LNI 192.168.206.53@tcp [ 670.558716] LNet: 41026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.168403] Key type .llcrypt unregistered [ 671.174333] Key type ._llcrypt unregistered [ 682.260593] Key type ._llcrypt registered [ 682.263114] Key type .llcrypt registered [ 683.923440] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:21:21 (1790770881) [ 685.433414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg653-server [ 687.126989] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:21:24 (1790770884) [ 687.679392] Key type .llcrypt unregistered [ 687.684194] Key type ._llcrypt unregistered [ 694.761170] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 694.945156] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.952276] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.958269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 697.620262] Key type ._llcrypt registered [ 697.622422] Key type .llcrypt registered [ 697.855921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 697.874750] alg: No test for adler32 (adler32-zlib) [ 698.844477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.934182] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 699.717714] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 701.725433] LNet: 42863:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.732611] LNetError: 42863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.742664] LNet: Removed LNI 192.168.206.53@tcp1 [ 701.756792] LNet: Removed LNI 10.1.2.3@tcp2 [ 702.356550] Key type .llcrypt unregistered [ 702.361709] Key type ._llcrypt unregistered [ 713.826606] Key type ._llcrypt registered [ 713.829186] Key type .llcrypt registered [ 715.085675] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:21:52 (1790770912) [ 715.741550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 715.758081] alg: No test for adler32 (adler32-zlib) [ 716.666730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.723395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.768958] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 717.568335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.585694] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.596764] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.609582] LNet: Removed LNI 192.168.206.53@tcp [ 717.643631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.696624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 717.736883] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 724.563445] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:22:02 (1790770922) [ 724.959335] LNet: 44422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.967559] LNetError: 44422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.981295] LNet: Removed LNI 192.168.206.53@tcp [ 725.732622] Key type .llcrypt unregistered [ 725.742560] Key type ._llcrypt unregistered [ 731.973452] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.041306] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.045455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.048531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.413412] Key type ._llcrypt registered [ 733.415870] Key type .llcrypt registered [ 733.557717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.568288] alg: No test for adler32 (adler32-zlib) [ 734.462773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.518735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.544273] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 735.193233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 735.223549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.017694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.028728] LNet: 45154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.032705] LNetError: 45154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 737.036620] LNet: Removed LNI 10.1.2.3@tcp [ 737.046384] LNet: Removed LNI 192.168.206.53@tcp [ 737.075147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.143646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 737.159402] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 737.162938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.739815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.763395] LNet: 45181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.770640] LNetError: 45181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 737.776603] LNetError: 45181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 737.783169] LNet: Removed LNI 10.1.2.3@tcp [ 737.791657] LNet: Removed LNI 192.168.206.53@tcp [ 737.813895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.880584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 737.912096] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 737.923509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.210468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 738.237362] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 739.334567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.359936] LNet: 45344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.365392] LNetError: 45344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.370917] LNetError: 45344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 739.377229] LNet: Removed LNI 10.1.2.3@tcp [ 739.379894] LNet: Removed LNI 192.168.206.53@tcp [ 739.395206] LNet: Removed LNI 192.168.206.53@tcp2 [ 739.419707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.472391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 739.494876] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 739.502510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.509202] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 747.089247] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:22:24 (1790770944) [ 747.351600] LNet: 45938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.359251] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 747.370169] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 747.381230] LNet: Removed LNI 10.1.2.3@tcp [ 747.384594] LNet: Removed LNI 192.168.206.53@tcp [ 747.415987] LNet: Removed LNI 192.168.206.53@tcp2 [ 748.624971] Key type .llcrypt unregistered [ 748.630759] Key type ._llcrypt unregistered [ 756.294252] Key type ._llcrypt registered [ 756.298539] Key type .llcrypt registered [ 756.644696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.680384] alg: No test for adler32 (adler32-zlib) [ 757.618697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.710668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.742828] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 758.751443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.773604] LNet: 46336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.778477] LNetError: 46336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.788615] LNet: Removed LNI 192.168.206.53@tcp [ 758.804388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.865817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 758.900741] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 759.580894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.620841] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.658341] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.679657] LNet: Removed LNI 192.168.206.53@tcp [ 759.721908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.822418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 759.875537] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 760.716916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.755250] LNet: 46396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.763897] LNetError: 46396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.789652] LNet: Removed LNI 192.168.206.53@tcp [ 760.837519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.965492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 761.022374] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 761.825465] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 762.263270] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.284395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.308975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.900348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.930933] LNet: 46479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.939694] LNetError: 46479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.965446] LNet: Removed LNI 192.168.206.53@tcp [ 765.001753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.101342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 765.135839] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 772.275278] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:22:49 (1790770969) [ 772.418738] LNet: 47071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.437479] LNetError: 47071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.452697] LNet: Removed LNI 192.168.206.53@tcp [ 772.911856] Key type .llcrypt unregistered [ 772.914368] Key type ._llcrypt unregistered [ 778.207418] Key type ._llcrypt registered [ 778.212601] Key type .llcrypt registered [ 778.513513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.530165] alg: No test for adler32 (adler32-zlib) [ 779.464099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.531350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.575362] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 780.514150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.541770] LNet: 47475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.548658] LNetError: 47475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.584308] LNet: Removed LNI 192.168.206.53@tcp [ 780.626887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.724985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 780.766789] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 787.962458] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:23:05 (1790770985) [ 788.160804] LNet: 48061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.171691] LNetError: 48061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.198470] LNet: Removed LNI 192.168.206.53@tcp [ 788.836616] Key type .llcrypt unregistered [ 788.839607] Key type ._llcrypt unregistered [ 795.251456] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.609139] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.615975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.620720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.152613] Key type ._llcrypt registered [ 798.158857] Key type .llcrypt registered [ 798.399262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.424230] alg: No test for adler32 (adler32-zlib) [ 799.397991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.461476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 799.484115] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 800.469917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.503717] LNet: 48516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.510215] LNetError: 48516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.520935] LNet: Removed LNI 192.168.206.53@tcp [ 800.557669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.624428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 800.661281] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 807.303722] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:23:25 (1790771005) [ 807.592696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.627086] LNet: 49114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.639767] LNetError: 49114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.662291] LNet: Removed LNI 192.168.206.53@tcp [ 807.700807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.843632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 814.110263] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:23:31 (1790771011) [ 814.320813] LNet: 49695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.896897] Key type .llcrypt unregistered [ 814.900210] Key type ._llcrypt unregistered [ 820.731865] Key type ._llcrypt registered [ 820.736052] Key type .llcrypt registered [ 820.962579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.984603] alg: No test for adler32 (adler32-zlib) [ 821.887221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.938805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.971388] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 822.808740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 822.825693] LNet: 50095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.833127] LNetError: 50095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.854103] LNet: Removed LNI 192.168.206.53@tcp [ 822.882088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.966366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 822.997707] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 829.074508] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:23:46 (1790771026) [ 829.290226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.306528] LNet: 50688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.311495] LNetError: 50688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.324796] LNet: Removed LNI 192.168.206.53@tcp [ 829.351383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.404268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 834.961788] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:23:52 (1790771032) [ 836.448968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 838.087152] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:23:55 (1790771035) [ 839.520151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 841.152396] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:23:59 (1790771039) [ 842.684257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 844.160689] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:24:02 (1790771042) [ 845.429653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 846.848784] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:24:04 (1790771044) [ 848.050790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 849.505601] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 08:24:07 (1790771047) [ 849.759905] LNet: 52709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.351898] Key type .llcrypt unregistered [ 850.354575] Key type ._llcrypt unregistered [ 856.342555] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 856.443944] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 856.450629] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 856.454356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.989983] Key type ._llcrypt registered [ 858.992076] Key type .llcrypt registered [ 859.147575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 859.156836] alg: No test for adler32 (adler32-zlib) [ 860.012781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.092790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 860.107679] LNet: Added LNI 192.168.206.53@tcp [8/512/0/180] [ 860.112237] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 860.113709] LNet: 53028:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 860.183366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.203360] LNet: 53047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.207840] LNetError: 53047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 860.212363] LNet: Removed LNI 10.1.2.3@tcp [ 860.214722] LNetError: 53047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.231962] LNet: Removed LNI 192.168.206.53@tcp [ 860.247207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.309550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 860.332977] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 860.394846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.415950] LNet: 53071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.423120] LNetError: 53071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.441114] LNet: Removed LNI 192.168.206.53@tcp [ 860.460346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.527666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 860.562735] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 860.567343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.572172] LNet: 53079:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 860.646974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.665036] LNet: 53097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.671634] LNetError: 53097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 860.676369] LNet: Removed LNI 10.1.2.3@tcp [ 860.679586] LNetError: 53097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.690705] LNet: Removed LNI 192.168.206.53@tcp [ 860.712974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.779995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 860.808750] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 860.811560] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.813235] LNet: 53105:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 860.959431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.976540] LNet: 53131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.982725] LNetError: 53131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 860.993145] LNet: Removed LNI 10.1.2.3@tcp [ 860.995750] LNetError: 53131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.011131] LNet: Removed LNI 192.168.206.53@tcp [ 861.035593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.089857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 861.106367] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 861.109798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 861.167223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.197869] LNet: 53159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.202489] LNetError: 53159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.212933] LNet: Removed LNI 10.1.2.3@tcp [ 861.215311] LNetError: 53159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.229920] LNet: Removed LNI 192.168.206.53@tcp [ 861.249594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.299195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 861.320945] LNet: Added LNI 192.168.206.53@tcp [8/256/0/182] [ 861.325793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 861.327799] LNet: 53167:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 861.390347] LNet: Removed LNI 10.1.2.3@tcp [ 861.404794] LNet: Removed LNI 192.168.206.53@tcp [ 861.926142] Key type .llcrypt unregistered [ 861.929580] Key type ._llcrypt unregistered [ 871.201300] Key type ._llcrypt registered [ 871.203709] Key type .llcrypt registered [ 872.173632] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 08:24:30 (1790771070) [ 873.484538] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 874.774723] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:24:32 (1790771072) [ 875.153474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 875.168222] alg: No test for adler32 (adler32-zlib) [ 876.110710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.168452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 876.188567] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 876.862285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.53 [ 876.882097] LNet: 54429:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 876.889868] LNet: Removed LNI 192.168.206.53@tcp [ 881.805511] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:24:39 (1790771079) [ 881.985441] LNet: 54999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.992911] LNetError: 54999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.005751] LNet: Removed LNI 192.168.206.53@tcp [ 882.536864] Key type .llcrypt unregistered [ 882.540481] Key type ._llcrypt unregistered [ 888.127098] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 888.355056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.939669] Key type ._llcrypt registered [ 890.942566] Key type .llcrypt registered [ 891.132890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.157944] alg: No test for adler32 (adler32-zlib) [ 892.378684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 897.365105] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:24:55 (1790771095) [ 897.616631] LNet: 55911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.628860] LNetError: 55911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 897.649370] LNet: Removed LNI 10.1.2.3@tcp [ 898.165388] Key type .llcrypt unregistered [ 898.171058] Key type ._llcrypt unregistered [ 903.852751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 904.127646] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 904.135344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 904.144955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.785855] Key type ._llcrypt registered [ 906.789226] Key type .llcrypt registered [ 906.937823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.951808] alg: No test for adler32 (adler32-zlib) [ 907.894095] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 913.442305] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:25:11 (1790771111) [ 914.846746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 916.317521] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:25:14 (1790771114) [ 916.496719] LNet: 57090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.502970] LNetError: 57090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 916.536281] LNet: Removed LNI 10.1.2.3@tcp [ 917.150552] Key type .llcrypt unregistered [ 917.152432] Key type ._llcrypt unregistered [ 922.841901] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 923.145683] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 923.152494] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 923.158626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 925.789394] Key type ._llcrypt registered [ 925.791143] Key type .llcrypt registered [ 925.925269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.951648] alg: No test for adler32 (adler32-zlib) [ 926.991638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.081768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.814226] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:25:34 (1790771134) [ 937.589301] LNet: 58326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.603094] LNetError: 58326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.615501] LNet: Removed LNI 10.1.2.3@tcp [ 938.129222] Key type .llcrypt unregistered [ 938.131475] Key type ._llcrypt unregistered [ 945.349117] Key type ._llcrypt registered [ 945.358314] Key type .llcrypt registered [ 945.742489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.774364] alg: No test for adler32 (adler32-zlib) [ 946.688706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 946.746268] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 955.009589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 955.662969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 955.713141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 955.782190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 955.814509] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 956.001373] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 956.202809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 956.228830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 956.258249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 956.298975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 956.325681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 956.369449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 956.388267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 956.431610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 956.481254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 956.497596] LNet: 58765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 956.516483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 956.553366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 956.606583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 956.663189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 956.720323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 956.787867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 956.841819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 957.357528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 957.384371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 957.409851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 957.429662] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 957.435919] LNet: Skipped 1 previous similar message [ 957.604495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.624307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 957.644447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.673867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.709563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 957.774402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.817679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.874439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.928448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.960271] LNet: 58876:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.986505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 958.051858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 958.096308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 958.122064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 958.149345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 958.188272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 958.231250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 958.678591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 958.710481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 958.758970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 958.775584] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 958.986259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 959.016343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 959.055322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 959.090756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 959.135976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 959.167883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 959.197448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.228469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.258727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 959.275609] LNet: 58987:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 959.295462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.322895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.365832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.390365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.421520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.462869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.532439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 959.982091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 960.029957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 960.090265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 960.307869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 960.335356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 960.366549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 960.403036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 960.439079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 960.482561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 960.520816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 960.560722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 960.601312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.626263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.655444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.686218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.710784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.739783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.789028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.819871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 961.385645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 961.425080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 961.471375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 961.509542] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 961.521547] LNet: Skipped 2 previous similar messages [ 961.758510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.792740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 961.843988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.884323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.917737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 961.948291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.982801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.018410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.063986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 962.087498] LNet: 59209:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 962.098809] LNet: 59209:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 962.129258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 962.198576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 962.247817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 962.299951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 962.348577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 962.397129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 962.449430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 962.970384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 963.020963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 963.097078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 963.375179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 963.406549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 963.446753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 963.476513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 963.522209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 963.549858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 963.582151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 963.612887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 963.649247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.693270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.746166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.781578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.816243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.844844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.869446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.898802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 964.157967] LNet: 59362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.165124] LNetError: 59362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.184168] LNet: Removed LNI 192.168.206.53@tcp [ 964.554445] Key type .llcrypt unregistered [ 964.556461] Key type ._llcrypt unregistered [ 980.817101] Key type ._llcrypt registered [ 980.822893] Key type .llcrypt registered [ 981.891493] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:26:19 (1790771179) [ 982.968437] Key type .llcrypt unregistered [ 982.974662] Key type ._llcrypt unregistered [ 989.690264] Key type ._llcrypt registered [ 989.693090] Key type .llcrypt registered [ 989.971530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.986628] alg: No test for adler32 (adler32-zlib) [ 990.958975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 990.988303] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1000.146838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1012.187442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.253356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1012.279147] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1013.067131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1013.544676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 1013.575758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 1 [ 1013.611424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 1013.654376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 1 [ 1013.695590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_interrupt -r 1 [ 1013.729600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 1 [ 1013.773269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_interrupt -r 1 [ 1013.805628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 1 [ 1013.853462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1013.875861] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1013.885969] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp1 [ 1014.050946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.074845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1014.100445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.124363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.148123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1014.177237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.222824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.255873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.285457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.306381] LNet: 60836:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.322979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.361974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.385951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.407964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.444142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.481628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.508578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1015.013895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 1015.044941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 1 [ 1015.075344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 1015.105099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 1 [ 1015.151390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_dropped -r 1 [ 1015.181440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 1 [ 1015.218587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_dropped -r 1 [ 1015.245239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 1 [ 1015.272223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1015.285750] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1015.290151] LNet: Skipped 2 previous similar messages [ 1015.475560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.516676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1015.548918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.578175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.606334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1015.631985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.660513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.685631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.715623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.728353] LNet: 60959:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.742979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.770444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.796879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.820307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.841547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.861075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.892405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1016.270056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 1016.298508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 1 [ 1016.326548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 1016.356822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 1 [ 1016.384802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_aborted -r 1 [ 1016.411283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 1 [ 1016.441301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_aborted -r 1 [ 1016.469833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 1 [ 1016.503147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1016.525419] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1016.532741] LNet: Skipped 2 previous similar messages [ 1016.691647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.713618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.739538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.759568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.794619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1016.838874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.865773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.894718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.921577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.936309] LNet: 61082:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.953324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.998903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.031732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.059471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.088947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.115358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.148379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1017.562144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 1017.594488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 1 [ 1017.628873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 1017.677317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 1 [ 1017.718963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_no_route -r 1 [ 1017.765922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 1 [ 1017.795680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_no_route -r 1 [ 1017.824111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 1 [ 1017.859530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1018.025926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.060850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1018.090546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1018.118400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1018.149818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1018.174602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1018.208397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1018.257348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1018.295223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1018.336815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.364370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.383984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.411560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.437234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.459591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.482692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.906599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 1018.932799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 1 [ 1018.960267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 1018.989876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 1 [ 1019.016419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_timeout -r 1 [ 1019.042401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 1 [ 1019.083219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_timeout -r 1 [ 1019.128835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 1 [ 1019.155238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1019.179692] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1019.186577] LNet: Skipped 5 previous similar messages [ 1019.404605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.438597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.467533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.496237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.521249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1019.565786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.600062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.633329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.663655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.686046] LNet: 61329:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.694494] LNet: 61329:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1019.714928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.746243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.790802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.827275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.855820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.881460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.908357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1020.325097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 1020.354519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_error -r 1 [ 1020.388642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 1020.419970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_error -r 1 [ 1020.443734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_error -r 1 [ 1020.470575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_error -r 1 [ 1020.506400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_error -r 1 [ 1020.537352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_error -r 1 [ 1020.564207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1020.759360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.787553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.813777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.835287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1020.866121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1020.891939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1020.913517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1020.939915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1020.972856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1020.998822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.018544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.040659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.064940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.082905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.122971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.156694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1025.823834] LNet: 61527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.831672] LNetError: 61527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1025.838988] LNet: Removed LNI 192.168.206.53@tcp [ 1025.859116] LNet: Removed LNI 192.168.206.53@tcp1 [ 1026.123541] Key type .llcrypt unregistered [ 1026.126496] Key type ._llcrypt unregistered [ 1041.119979] Key type ._llcrypt registered [ 1041.125119] Key type .llcrypt registered [ 1042.347590] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:27:20 (1790771240) [ 1043.556369] Key type .llcrypt unregistered [ 1043.560695] Key type ._llcrypt unregistered [ 1050.210785] Key type ._llcrypt registered [ 1050.214607] Key type .llcrypt registered [ 1050.540942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.572688] alg: No test for adler32 (adler32-zlib) [ 1051.555245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1051.602330] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1060.412310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1061.518298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1061.593141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1061.664605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1061.699838] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1062.089802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.126942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.161640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.226407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.296446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1062.361458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.458600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.522875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.602324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.640593] LNet: 62779:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.696514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.754665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.807282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.911503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.978908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.022633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.086519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1063.763915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1063.805598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1063.854799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1063.878666] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1064.096585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.133600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1064.168882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.225892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.303204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1064.411879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.470770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.536625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.605669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.632588] LNet: 62890:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.669914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.745806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.782682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.834876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.887851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.934960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.967579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.476972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1065.510234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1065.553137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1065.585974] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1065.876640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.911517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.940754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.965650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.027472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1066.089612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.147946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.191326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.251576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.282240] LNet: 63001:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.311670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.350727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.400767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.430217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.464720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.499790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.571646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1066.932852] LNet: 63042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.947614] LNetError: 63042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.986659] LNet: Removed LNI 192.168.206.53@tcp [ 1067.388160] Key type .llcrypt unregistered [ 1067.389642] Key type ._llcrypt unregistered [ 1084.727794] Key type ._llcrypt registered [ 1084.730162] Key type .llcrypt registered [ 1085.763019] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:28:03 (1790771283) [ 1086.615425] Key type .llcrypt unregistered [ 1086.617965] Key type ._llcrypt unregistered [ 1093.938187] Key type ._llcrypt registered [ 1093.941459] Key type .llcrypt registered [ 1094.128597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.149892] alg: No test for adler32 (adler32-zlib) [ 1095.060456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1095.093302] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1101.940723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1111.336251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.381820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1111.401181] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1112.241380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1112.807091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1112.848879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 1 [ 1112.883915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1112.915630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 1 [ 1112.948376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_dropped -r 1 [ 1112.979241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 1 [ 1113.024766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_dropped -r 1 [ 1113.059106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 1 [ 1113.096463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1113.115836] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1113.121620] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp1 [ 1113.302553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.328251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.356997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.392612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.440371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1113.481834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.511403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.537528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.563420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.579841] LNet: 64515:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.595835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.627845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.669829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.705640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.732619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.758816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.796813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.853581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.441775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1114.487472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_error -r 1 [ 1114.529424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1114.560200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_error -r 1 [ 1114.601389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_error -r 1 [ 1114.638547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_error -r 1 [ 1114.680270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_error -r 1 [ 1114.724039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_error -r 1 [ 1114.765263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1114.779724] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1114.787269] LNet: Skipped 2 previous similar messages [ 1115.007588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.058249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.099444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.135605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.165907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1115.194643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.222900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.250390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.272384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.289729] LNet: 64640:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.304608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.326728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.352729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.384886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.411659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.444802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.477298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.494500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.943213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1115.968537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_timeout -r 1 [ 1115.992698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1116.016733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_timeout -r 1 [ 1116.045687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_timeout -r 1 [ 1116.069859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_timeout -r 1 [ 1116.099267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_timeout -r 1 [ 1116.132368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_timeout -r 1 [ 1116.186552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.153@tcp [ 1116.217250] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1116.223511] LNet: Skipped 1 previous similar message [ 1116.477425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.510426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.543692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.584869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.618352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1116.664099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.715478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.771237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.838392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.867824] LNet: 64765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.896996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.935665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.973727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.997464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.023641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.056646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.092749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1117.122636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.180630] LNet: 64843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.191799] LNetError: 64843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.207078] LNet: Removed LNI 192.168.206.53@tcp [ 1122.222326] LNet: Removed LNI 192.168.206.53@tcp1 [ 1122.672373] Key type .llcrypt unregistered [ 1122.679268] Key type ._llcrypt unregistered [ 1138.065235] Key type ._llcrypt registered [ 1138.067963] Key type .llcrypt registered [ 1139.221458] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:28:57 (1790771337) [ 1140.859132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1142.326623] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:29:00 (1790771340) [ 1143.094766] Key type .llcrypt unregistered [ 1143.101598] Key type ._llcrypt unregistered [ 1148.555151] Key type ._llcrypt registered [ 1148.559509] Key type .llcrypt registered [ 1148.790073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.815395] alg: No test for adler32 (adler32-zlib) [ 1149.680485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.703558] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1157.138102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1157.698768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1157.753750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1157.798401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1157.852366] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1157.926204] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1158.068590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1158.103511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1158.145809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1158.183098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1158.255501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1158.312393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1158.347116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1158.379673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1158.404738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1158.425988] LNet: 66385:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1158.452583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1158.498360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1158.540549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.571236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1158.622239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1158.655376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1158.897327] LNet: 66424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.902452] LNetError: 66424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.916070] LNet: Removed LNI 192.168.206.53@tcp [ 1159.250785] Key type .llcrypt unregistered [ 1159.255136] Key type ._llcrypt unregistered [ 1173.721447] Key type ._llcrypt registered [ 1173.724734] Key type .llcrypt registered [ 1173.871242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1173.903229] alg: No test for adler32 (adler32-zlib) [ 1174.799074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1174.830239] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1182.031274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1190.954384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.002309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1191.024518] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1191.756637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1192.547524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1192.586992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e network_timeout -r 1 [ 1192.615654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1192.673353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e network_timeout -r 1 [ 1192.751364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e network_timeout -r 1 [ 1192.822443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e network_timeout -r 1 [ 1192.876133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e network_timeout -r 1 [ 1192.948881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e network_timeout -r 1 [ 1192.993685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1193.019943] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1193.207291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.234565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.262473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.289667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.311077] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp1 [ 1193.317037] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp1 [ 1193.323678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.349507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.374441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.408395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.460990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.484179] LNet: 67308:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.520215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.571559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.618542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.656956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.692041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.729401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.503275] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1199.752091] LNet: 67382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.756880] LNetError: 67382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.763232] LNet: Removed LNI 192.168.206.53@tcp [ 1199.775139] LNet: Removed LNI 192.168.206.53@tcp1 [ 1200.194418] Key type .llcrypt unregistered [ 1200.200022] Key type ._llcrypt unregistered [ 1215.935250] Key type ._llcrypt registered [ 1215.937378] Key type .llcrypt registered [ 1216.931477] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:30:14 (1790771414) [ 1217.212811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.233227] alg: No test for adler32 (adler32-zlib) [ 1218.126830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.170415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.186947] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1218.844643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1218.874530] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1219.734556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1219.780528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1219.827691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1219.858172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1219.892473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1219.930830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1219.980078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1220.010336] LNetError: 68480:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1220.191399] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1220.191739] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1222.239923] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1222.408973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1226.335447] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1227.040793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1227.082632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.129905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.159441] LNet: 68587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.165666] LNetError: 68587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.177314] LNet: Removed LNI 192.168.206.53@tcp [ 1227.187385] LNet: Removed LNI 192.168.206.53@tcp1 [ 1227.203529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.289805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1227.338181] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1227.908156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1227.928323] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1228.596200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1228.659472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.712881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1228.757666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1228.794699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1228.851113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1228.889125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1228.920299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1228.935866] LNetError: 68860:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1229.279443] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1232.387744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1235.423463] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1235.437440] LNet: Skipped 1 previous similar message [ 1240.274827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1240.354839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.435643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.115431] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:30:45 (1790771445) [ 1248.334607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1248.351277] LNet: 69578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.360383] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.368660] LNetError: 69578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1248.378167] LNet: Removed LNI 192.168.206.53@tcp [ 1248.392442] LNet: Removed LNI 192.168.206.53@tcp1 [ 1248.414690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.460673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1248.478819] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1249.035866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1249.050344] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1249.851975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1249.921871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1249.958108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1249.995351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1250.031690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1250.064661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1250.084420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 0 [ 1250.162574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1250.527276] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.53@tcp [ 1251.551565] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1251.556766] LNet: Skipped 1 previous similar message [ 1266.440474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1266.610592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1266.633755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.654991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 500 [ 1266.731893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1272.261982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1272.336509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.359734] LNet: 70070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.364903] LNetError: 70070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.370991] LNetError: 70070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1272.379438] LNet: Removed LNI 192.168.206.53@tcp [ 1272.387421] LNet: Removed LNI 192.168.206.53@tcp1 [ 1272.412951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.466113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1272.489199] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1273.193773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1273.226356] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1274.028772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1274.079638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.160800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1274.197571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1274.244408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1274.284155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1274.341078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1274.372764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.53@tcp --health 0 [ 1275.552424] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1275.566683] LNet: Skipped 3 previous similar messages [ 1275.743359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.061468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.087602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.108988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.898245] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:31:30 (1790771490) [ 1293.289922] LNet: 71128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.297345] LNetError: 71128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.302066] LNetError: 71128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1294.367206] LNet: Removed LNI 192.168.206.53@tcp [ 1294.383782] LNet: Removed LNI 192.168.206.53@tcp1 [ 1294.972444] Key type .llcrypt unregistered [ 1294.978131] Key type ._llcrypt unregistered [ 1300.031134] Key type ._llcrypt registered [ 1300.033105] Key type .llcrypt registered [ 1300.209236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1300.225436] alg: No test for adler32 (adler32-zlib) [ 1301.086201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1301.104875] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1308.149991] Lustre: DEBUG MARKER: Initial discovery [ 1308.202695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1309.870188] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1309.904620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1309.932342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1309.948516] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1309.960630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1309.993742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.153@tcp -r 1 -m GET -l 3 [ 1310.035302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.153@tcp -r 1 -m GET -e local_error [ 1310.062547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.153@tcp -r 1 -m PUT -l 6 [ 1310.303515] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1311.040400] Lustre: DEBUG MARKER: Force oleg653-client.virtnet to discover 192.168.206.153@tcp (in background) [ 1311.096879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.153@tcp [ 1312.048356] Lustre: DEBUG MARKER: Force oleg653-server to discover 192.168.206.53@tcp [ 1313.812673] Lustre: DEBUG MARKER: Wait for 71879 [ 1314.143302] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1315.271886] Lustre: DEBUG MARKER: Finished wait on 71879 [ 1315.419086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1315.456469] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1315.461687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1315.641974] LNet: 72338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.653570] LNetError: 72338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.676826] LNet: Removed LNI 192.168.206.53@tcp [ 1316.017667] Key type .llcrypt unregistered [ 1316.021273] Key type ._llcrypt unregistered [ 1331.792403] Key type ._llcrypt registered [ 1331.796683] Key type .llcrypt registered [ 1332.960757] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:32:10 (1790771530) [ 1333.703737] Key type .llcrypt unregistered [ 1333.706626] Key type ._llcrypt unregistered [ 1339.015295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1339.169711] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1339.177900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1339.182342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1341.694552] Key type ._llcrypt registered [ 1341.701190] Key type .llcrypt registered [ 1341.887083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1341.903533] alg: No test for adler32 (adler32-zlib) [ 1342.850268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.904040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1342.923888] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1343.693563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1343.737615] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1351.878788] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:32:29 (1790771549) [ 1352.167313] LNet: 74413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.172828] LNetError: 74413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1352.178280] LNet: Removed LNI 10.1.2.3@tcp [ 1352.190626] LNet: Removed LNI 192.168.206.53@tcp [ 1352.839448] Key type .llcrypt unregistered [ 1352.846883] Key type ._llcrypt unregistered [ 1359.311406] Key type ._llcrypt registered [ 1359.316398] Key type .llcrypt registered [ 1359.537853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.555524] alg: No test for adler32 (adler32-zlib) [ 1360.535502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.615834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.637256] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1361.343809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1361.374811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1362.277499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.53@tcp [ 1363.359405] LNetError: Unexpected error -22 connecting to 192.168.206.53@tcp at host 192.168.206.53:988 [ 1367.948575] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:32:45 (1790771565) [ 1368.048931] LNet: 75540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.062552] LNetError: 75540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1368.076067] LNet: Removed LNI 10.1.2.3@tcp [ 1368.095557] LNet: Removed LNI 192.168.206.53@tcp [ 1368.791474] Key type .llcrypt unregistered [ 1368.793470] Key type ._llcrypt unregistered [ 1374.440248] Key type ._llcrypt registered [ 1374.442848] Key type .llcrypt registered [ 1374.588800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1374.598567] alg: No test for adler32 (adler32-zlib) [ 1375.533886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.592470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.624583] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1376.220672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1376.236226] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 1376.967827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.53@tcp1 --nid 192.168.206.53@tcp2 [ 1380.370522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp1 192.168.206.53@tcp2 [ 1380.393260] LNetError: 76380:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.53@tcp1 to 192.168.206.53@tcp2 - Route table may be misconfigured [ 1380.421653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp2 192.168.206.53@tcp1 [ 1385.170281] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:33:03 (1790771583) [ 1385.304701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.328030] LNet: 76950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.350288] LNetError: 76950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.357688] LNet: Removed LNI 192.168.206.53@tcp1 [ 1386.408723] LNet: Removed LNI 192.168.206.53@tcp2 [ 1386.446348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.492147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.514239] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1387.204306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1387.226049] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1388.033697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1388.064432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.53@tcp [ 1388.097439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp1 192.168.206.53@tcp1 [ 1388.127828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.53@tcp -d 192.168.206.53@tcp -e network_timeout [ 1388.162972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -e network_timeout [ 1388.213880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -e network_timeout [ 1388.263578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -e network_timeout [ 1388.310288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1388.345077] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1388.390780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1388.509936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.511113] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1388.616606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1395.029812] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 08:33:12 (1790771592) [ 1395.110176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.140167] LNet: 77809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.156612] LNetError: 77809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.162374] LNetError: 77809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1395.168084] LNet: Removed LNI 192.168.206.53@tcp [ 1396.201259] LNet: Removed LNI 192.168.206.53@tcp1 [ 1396.228387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.315750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1396.542521] LNet: 77853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.866693] Key type .llcrypt unregistered [ 1396.868610] Key type ._llcrypt unregistered [ 1407.612780] Key type ._llcrypt registered [ 1407.614686] Key type .llcrypt registered [ 1408.868298] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 08:33:26 (1790771606) [ 1409.260496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.270878] alg: No test for adler32 (adler32-zlib) [ 1410.184497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.351283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1412.892690] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 08:33:30 (1790771610) [ 1413.165038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.185690] LNet: 78950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.204528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.247578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.274731] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1413.914293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.933645] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1414.714671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1414.745466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp1 [ 1414.772455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp1 [ 1420.727191] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 08:33:38 (1790771618) [ 1422.611128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg653-server [ 1424.134559] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 08:33:42 (1790771622) [ 1425.938588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg653-server [ 1427.453816] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 08:33:45 (1790771625) [ 1429.302173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg653-server [ 1430.902718] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 08:33:48 (1790771628) [ 1432.697711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg653-server [ 1434.525425] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 08:33:52 (1790771632) [ 1436.489981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg653-server [ 1438.360602] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 08:33:55 (1790771635) [ 1440.326278] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg653-server [ 1442.435547] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 08:33:59 (1790771639) [ 1443.938916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg653-server [ 1445.381330] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 08:34:03 (1790771643) [ 1447.045318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg653-server [ 1448.568621] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 08:34:06 (1790771646) [ 1450.255201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg653-server [ 1452.297263] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 08:34:09 (1790771649) [ 1452.408821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.444521] LNet: 82821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.454298] LNetError: 82821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.465569] LNet: Removed LNI 192.168.206.53@tcp [ 1453.481431] LNet: Removed LNI 192.168.206.53@tcp1 [ 1453.513952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.612075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.642027] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1454.429128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1454.530572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1454.792704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.814710] LNet: 82974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.822382] LNetError: 82974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.830697] LNetError: 82974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1455.855503] LNet: Removed LNI 192.168.206.53@tcp [ 1455.896865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.990969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1456.031106] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1456.811632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1456.971867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.992272] LNet: 83119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.998773] LNetError: 83119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.008141] LNet: Removed LNI 192.168.206.53@tcp [ 1457.029726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.079513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.102110] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1457.822172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1457.902282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1458.039418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.109702] LNet: Removed LNI 192.168.206.53@tcp [ 1459.141989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.198501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1459.221584] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1459.933794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1460.093314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.115090] LNetError: 83412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.125467] LNetError: 83412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1460.150323] LNet: Removed LNI 192.168.206.53@tcp [ 1460.172381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.232243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1460.257498] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1460.898382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1460.989903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1461.183511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.202681] LNet: 83559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.208622] LNet: 83559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1462.251643] LNet: Removed LNI 192.168.206.53@tcp [ 1462.285758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.368459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1462.386191] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1462.964420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1463.192481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.232750] LNet: Removed LNI 192.168.206.53@tcp [ 1464.273546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.364478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.414269] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1465.300985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1465.395966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1465.663568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.717435] LNetError: 83852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1465.724930] LNetError: 83852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1465.752673] LNet: Removed LNI 192.168.206.53@tcp [ 1465.773119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.836285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1465.853731] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1466.697551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1466.893943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.943698] LNet: Removed LNI 192.168.206.53@tcp [ 1467.972501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.013546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.041519] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1468.596605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1468.691961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1469.984914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.006265] LNet: 84154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.012786] LNet: 84154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1471.091025] LNet: Removed LNI 192.168.206.53@tcp [ 1471.145035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.238791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1471.258610] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1472.135652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1472.373712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.447730] LNet: Removed LNI 192.168.206.53@tcp [ 1473.490143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.538595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.554196] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1474.111702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1474.186737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1475.568957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.588186] LNetError: 84456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.596552] LNetError: 84456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1475.627534] LNet: Removed LNI 192.168.206.53@tcp [ 1475.644894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.687130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1475.702172] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1476.240582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1477.693573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.759234] LNet: Removed LNI 192.168.206.53@tcp [ 1478.791516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.854775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.883941] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1479.449471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1479.507972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.545277] LNet: Removed LNI 192.168.206.53@tcp [ 1479.570455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.614218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.638758] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1480.410271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1480.499202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.522968] LNet: Removed LNI 192.168.206.53@tcp [ 1480.539369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.589882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1480.671727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.723542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.826779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1480.937366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.007870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.103147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1481.206260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.276180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.330649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1481.373078] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1482.167966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.195131] LNet: Removed LNI 192.168.206.53@tcp [ 1482.216522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.282556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1488.399061] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:34:46 (1790771686) [ 1488.603571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.629334] LNet: 85631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.643041] LNet: 85631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1488.679696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.739859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1488.765813] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1489.938717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.967710] LNet: Removed LNI 192.168.206.53@tcp231 [ 1489.986293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.045774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1490.082211] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1490.089598] LNet: 85777:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1490.464684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1490.520278] LNet: Removed LNI 192.168.206.53@tcp231 [ 1490.540440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1490.565475] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1491.698172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.737669] LNetError: 85922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.746949] LNetError: 85922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1491.763564] LNet: Removed LNI 192.168.206.53@tcp231 [ 1491.791864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.860055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1491.880036] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/0] [ 1491.884714] LNet: 85930:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1492.355592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1492.397041] LNet: Removed LNI 192.168.206.53@tcp231 [ 1492.411666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1492.449978] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/0] [ 1493.590813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.628198] LNet: Removed LNI 192.168.206.53@tcp231 [ 1493.643600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.709562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1493.730496] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/60] [ 1493.739577] LNet: 86081:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1493.985740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1494.032683] LNet: Removed LNI 192.168.206.53@tcp231 [ 1494.052289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1494.075755] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/60] [ 1495.310738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.351793] LNet: Removed LNI 192.168.206.53@tcp231 [ 1495.382812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.482697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1495.511793] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1496.072621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1496.109656] LNet: Removed LNI 192.168.206.53@tcp231 [ 1496.129133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1496.166155] LNet: Added LNI 192.168.206.53@tcp231 [8/256/0/180] [ 1502.905694] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:35:00 (1790771700) [ 1503.158431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.201242] LNet: Removed LNI 192.168.206.53@tcp231 [ 1503.228612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.330720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1503.365535] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1504.041241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1504.074553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1504.100072] LNet: Removed LNI 192.168.206.53@tcp232 [ 1504.148590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1504.195159] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1504.928270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1504.980809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1505.013300] LNet: Removed LNI 192.168.206.53@tcp232 [ 1505.030144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1505.055902] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1505.789320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1505.832690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1505.886045] LNet: Removed LNI 192.168.206.53@tcp232 [ 1505.926499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1505.970126] LNet: Added LNI 192.168.206.53@tcp232 [8/256/0/180] [ 1506.985354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1507.035354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1507.093663] LNet: Removed LNI 192.168.206.53@tcp232 [ 1507.110812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1513.458757] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:35:11 (1790771711) [ 1514.521557] Key type .llcrypt unregistered [ 1514.524148] Key type ._llcrypt unregistered [ 1520.784332] Key type ._llcrypt registered [ 1520.785989] Key type .llcrypt registered [ 1520.999426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1521.012396] alg: No test for adler32 (adler32-zlib) [ 1521.969302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1521.985478] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1528.983489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1538.252346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.315124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1538.379652] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1539.197246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1539.916074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_interrupt -r 2 [ 1539.957619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 2 [ 1539.995798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_interrupt -r 2 [ 1540.040579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 2 [ 1540.084158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_interrupt -r 2 [ 1540.121366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_interrupt -r 2 [ 1540.167589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_interrupt -r 2 [ 1540.203612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_interrupt -r 2 [ 1540.251957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1540.276551] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1540.288552] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1540.501463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1540.542173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1540.578743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1540.611969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1540.643774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1540.674580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1540.715584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1540.758781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1540.789558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1540.809594] LNet: 88739:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1540.824195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1540.850542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1540.877430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.907702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.962636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.996579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1541.480451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_dropped -r 2 [ 1541.514978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 2 [ 1541.550666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_dropped -r 2 [ 1541.598285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 2 [ 1541.638851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_dropped -r 2 [ 1541.688902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_dropped -r 2 [ 1541.740425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_dropped -r 2 [ 1541.782083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_dropped -r 2 [ 1541.824145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1541.843606] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1542.084659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1542.117822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1542.159231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1542.191516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1542.219121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1542.251460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1542.303718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1542.346385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1542.388208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1542.418903] LNet: 88862:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1542.464523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1542.516679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1542.551817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1542.601335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1542.635956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1542.693845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1543.188187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_aborted -r 2 [ 1543.222450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 2 [ 1543.248314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_aborted -r 2 [ 1543.283383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 2 [ 1543.336660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_aborted -r 2 [ 1543.380621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_aborted -r 2 [ 1543.431864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_aborted -r 2 [ 1543.474870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_aborted -r 2 [ 1543.518832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1543.538537] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1543.760713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1543.791682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1543.817599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1543.852250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1543.887232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1543.920319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1543.944944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1543.972682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1543.998130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1544.014569] LNet: 88984:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1544.037070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1544.067276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1544.101513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1544.156697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1544.190529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1544.227107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1544.917241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_no_route -r 2 [ 1544.942598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 2 [ 1544.967825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_no_route -r 2 [ 1544.991380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 2 [ 1545.023697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_no_route -r 2 [ 1545.064531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_no_route -r 2 [ 1545.127983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_no_route -r 2 [ 1545.181180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_no_route -r 2 [ 1545.227547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1545.455779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1545.480935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1545.508675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1545.535643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1545.567607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1545.598448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1545.627079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1545.651632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1545.676566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1545.701545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1545.728490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1545.772569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1545.816841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1545.867421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1545.922079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1546.445775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e local_timeout -r 2 [ 1546.497756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 2 [ 1546.570361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e local_timeout -r 2 [ 1546.615809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 2 [ 1546.664499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e local_timeout -r 2 [ 1546.730194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e local_timeout -r 2 [ 1546.773146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e local_timeout -r 2 [ 1546.839730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e local_timeout -r 2 [ 1546.880844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1546.906175] LNet: There was an unexpected network error while writing to 192.168.206.153: rc = -22 [ 1546.910977] LNet: Skipped 1 previous similar message [ 1547.206366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1547.232332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1547.257832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1547.284959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1547.346844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1547.379479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1547.409413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1547.442774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1547.485849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1547.510107] LNet: 89229:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1547.515876] LNet: 89229:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1547.536211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1547.571101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1547.611872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1547.650929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1547.685219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1547.721430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1548.215471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp -m GET -e remote_dropped -r 2 [ 1548.247223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 2 [ 1548.276870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp -m GET -e remote_dropped -r 2 [ 1548.324168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 2 [ 1548.362959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp -m GET -e remote_dropped -r 2 [ 1548.417921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.153@tcp1 -m GET -e remote_dropped -r 2 [ 1548.458126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp -m GET -e remote_dropped -r 2 [ 1548.494503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.53@tcp1 -d 192.168.206.53@tcp1 -m GET -e remote_dropped -r 2 [ 1548.544294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.53@tcp 192.168.206.153@tcp [ 1548.566535] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 1548.872584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1548.917554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1548.947179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1549.004581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1549.037792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1549.078115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1549.125246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1549.159293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1549.192851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1549.221481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1549.243194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1549.268372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.298767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.335369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1549.367698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1554.604703] LNet: 89425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.609975] LNetError: 89425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.616289] LNet: Removed LNI 192.168.206.53@tcp [ 1554.628372] LNet: Removed LNI 192.168.206.53@tcp1 [ 1555.011959] Key type .llcrypt unregistered [ 1555.014873] Key type ._llcrypt unregistered [ 1570.649202] Key type ._llcrypt registered [ 1570.654374] Key type .llcrypt registered [ 1571.733246] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:36:09 (1790771769) [ 1572.079039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1572.094683] alg: No test for adler32 (adler32-zlib) [ 1572.992596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.029120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.046743] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1573.751531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1573.813956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.53@tcp [ 1573.845424] LNetError: 90404:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1573.985660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.53@tcp [ 1574.132696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.765808] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:36:18 (1790771778) [ 1580.863449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.895306] LNet: 90987:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.905982] LNetError: 90987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.924668] LNet: Removed LNI 192.168.206.53@tcp [ 1580.941957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.030254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.082837] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1582.048434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1582.070369] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1582.928657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.53@tcp [ 1582.952284] LNetError: 91260:0:(api-ni.c:4483:lnet_ni_set_healthv()) manually adding local NI 192.168.206.53@tcp to recovery [ 1583.071488] LNet: 1 local NIs in recovery (showing 1): 192.168.206.53@tcp [ 1583.073825] LNet: There was an unexpected network error while writing to 192.168.206.53: rc = -22 [ 1584.221985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.53@tcp [ 1591.566370] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:36:29 (1790771789) [ 1593.365815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg653-server [ 1595.038612] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:36:32 (1790771792) [ 1597.166218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg653-server [ 1599.103462] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:36:36 (1790771796) [ 1601.115420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg653-server [ 1603.035959] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:36:40 (1790771800) [ 1603.214109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.230911] LNet: 92878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1603.243183] LNetError: 92878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1603.254318] LNet: Removed LNI 192.168.206.53@tcp [ 1603.262183] LNet: Removed LNI 192.168.206.53@tcp1 [ 1603.277459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.352484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.396667] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1604.296631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1604.400162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1604.478110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1604.632348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1604.684442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1604.802543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1612.397351] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:36:49 (1790771809) [ 1612.483127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.521289] LNet: 93608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.531791] LNetError: 93608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.541704] LNetError: 93608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.568803] LNet: Removed LNI 192.168.206.53@tcp [ 1612.597323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.716640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.756888] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1621.209986] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:36:58 (1790771818) [ 1621.523875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.580427] LNet: 94317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.586392] LNetError: 94317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.619400] LNet: Removed LNI 192.168.206.53@tcp [ 1621.667926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.761677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.797428] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1628.952927] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:37:06 (1790771826) [ 1630.612912] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1632.500453] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:37:10 (1790771830) [ 1632.836309] LNet: 95353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.841901] LNetError: 95353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.858361] LNet: Removed LNI 192.168.206.53@tcp [ 1633.519054] Key type .llcrypt unregistered [ 1633.520888] Key type ._llcrypt unregistered [ 1640.630931] Key type ._llcrypt registered [ 1640.640884] Key type .llcrypt registered [ 1640.974791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.997204] alg: No test for adler32 (adler32-zlib) [ 1641.948960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1641.997679] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1651.396694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1656.852365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.153@tcp [ 1657.397995] LNet: 95771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.406948] LNetError: 95771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.426531] LNet: Removed LNI 192.168.206.53@tcp [ 1658.116495] Key type .llcrypt unregistered [ 1658.121804] Key type ._llcrypt unregistered [ 1670.279167] Key type ._llcrypt registered [ 1670.283232] Key type .llcrypt registered [ 1671.510762] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1673.682937] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1676.560258] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:37:53 (1790771873) [ 1677.895709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1680.137267] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:37:57 (1790771877) [ 1682.624339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg653-server [ 1684.547953] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:38:02 (1790771882) [ 1686.384289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg653-server [ 1687.950346] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:38:05 (1790771885) [ 1689.681316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1689.728281] alg: No test for adler32 (adler32-zlib) [ 1690.824670] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 1691.099481] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1692.759144] Key type lgssc registered [ 1693.785211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1713.493409] Key type lgssc unregistered [ 1713.830659] LNet: 98955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.846454] LNetError: 98955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.873781] LNet: Removed LNI 192.168.206.53@tcp [ 1714.702868] Key type .llcrypt unregistered [ 1714.709790] Key type ._llcrypt unregistered [ 1732.637907] Key type ._llcrypt registered [ 1732.640933] Key type .llcrypt registered [ 1733.854838] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:38:51 (1790771931) [ 1734.206371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1734.216880] alg: No test for adler32 (adler32-zlib) [ 1735.097523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.189087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1735.242771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1735.284671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1735.328979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1741.189839] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:38:58 (1790771938) [ 1741.702507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.775356] LNet: 100379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.808888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.858435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.880612] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1742.729678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.755272] LNet: 100523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.759684] LNetError: 100523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.775418] LNet: Removed LNI 192.168.206.53@tcp [ 1742.803773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.847306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1742.872209] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1743.427992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.459760] LNetError: 100551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.479050] LNet: Removed LNI 192.168.206.53@tcp [ 1743.500719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.572723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1743.618801] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1751.495878] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:39:08 (1790771948) [ 1751.573323] LNet: 101133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.582957] LNet: 101133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1751.592383] LNetError: 101133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1751.615259] LNet: Removed LNI 192.168.206.53@tcp [ 1752.265818] Key type .llcrypt unregistered [ 1752.270807] Key type ._llcrypt unregistered [ 1759.489695] Key type ._llcrypt registered [ 1759.492798] Key type .llcrypt registered [ 1759.791686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1759.804637] alg: No test for adler32 (adler32-zlib) [ 1760.820835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.912526] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1760.915534] LNet: *** cfs_fail_loc=e003, val=100*** [ 1761.646817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1761.667440] LNet: *** cfs_fail_loc=e003, val=100*** [ 1761.672960] LNet: Skipped 2 previous similar messages [ 1761.683514] Ping reply received with mapping: [ 1761.685460] found metadata reply NI[192.168.206.53@tcp] [ 1761.688481] 35 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1761.698680] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1761.702416] 5a 5a 5a 5a 5a 5a 5a 5a [ 1761.865343] LNet: 101538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.872282] LNetError: 101538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.902468] LNet: Removed LNI 192.168.206.53@tcp [ 1762.430102] Key type .llcrypt unregistered [ 1762.435723] Key type ._llcrypt unregistered [ 1775.786594] Key type ._llcrypt registered [ 1775.793215] Key type .llcrypt registered [ 1777.530460] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:39:34 (1790771974) [ 1778.109364] Key type .llcrypt unregistered [ 1778.111304] Key type ._llcrypt unregistered [ 1785.339392] Key type ._llcrypt registered [ 1785.341376] Key type .llcrypt registered [ 1785.607997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.640210] alg: No test for adler32 (adler32-zlib) [ 1786.721038] LNetError: 102579:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1787.112743] Key type .llcrypt unregistered [ 1787.119895] Key type ._llcrypt unregistered [ 1792.078399] Key type ._llcrypt registered [ 1792.084747] Key type .llcrypt registered [ 1793.346786] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:39:50 (1790771990) [ 1793.861415] Key type .llcrypt unregistered [ 1793.864820] Key type ._llcrypt unregistered [ 1799.826308] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1799.829926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1802.447334] Key type ._llcrypt registered [ 1802.456386] Key type .llcrypt registered [ 1802.760252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1802.784313] alg: No test for adler32 (adler32-zlib) [ 1803.785903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.858350] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1813.882300] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:40:11 (1790772011) [ 1814.237444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.271931] LNet: 104357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.285356] LNetError: 104357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1814.310680] LNet: Removed LNI 10.1.2.3@tcp [ 1814.357493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.162727] Key type .llcrypt unregistered [ 1815.165450] Key type ._llcrypt unregistered [ 1821.452278] Key type ._llcrypt registered [ 1821.454062] Key type .llcrypt registered [ 1821.725419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1821.748604] alg: No test for adler32 (adler32-zlib) [ 1822.640920] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1822.689706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1822.813544] LNet: 104644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.821904] LNetError: 104644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.837566] LNet: Removed LNI 192.168.206.53@tcp [ 1823.423379] Key type .llcrypt unregistered [ 1823.426462] Key type ._llcrypt unregistered [ 1833.624838] Key type ._llcrypt registered [ 1833.627300] Key type .llcrypt registered [ 1834.836916] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:40:32 (1790772032) [ 1835.418499] Key type .llcrypt unregistered [ 1835.422615] Key type ._llcrypt unregistered [ 1842.488449] Key type ._llcrypt registered [ 1842.493280] Key type .llcrypt registered [ 1842.816878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1842.837088] alg: No test for adler32 (adler32-zlib) [ 1850.256850] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:40:47 (1790772047) [ 1850.416310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.459930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.521638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.549833] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1851.225841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.257870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1851.284701] LNet: Removed LNI 192.168.206.53@tcp [ 1851.526439] LNet: 106386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.888743] Key type .llcrypt unregistered [ 1851.890497] Key type ._llcrypt unregistered [ 1863.678373] Key type ._llcrypt registered [ 1863.680911] Key type .llcrypt registered [ 1865.095461] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:41:02 (1790772062) [ 1865.517336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.538975] alg: No test for adler32 (adler32-zlib) [ 1866.426749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.488770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1866.515468] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1867.350202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1867.388294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.53@tcp [ 1867.431938] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1867.445043] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1867.449500] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1867, 0, 0 [ 1867.460165] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1867.464403] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1867.468142] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1867.472685] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1867, 0, 0 [ 1867.480694] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1867.485739] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1867.499469] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1867.500955] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1867, 0, 1917 [ 1867.506312] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1867.509144] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1867.516666] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1867.523942] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1867, 0, 1917 [ 1867.536425] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1873.820884] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:41:11 (1790772071) [ 1874.316695] LNet: 107924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.322798] LNetError: 107924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.373550] LNet: Removed LNI 192.168.206.53@tcp [ 1876.115386] Key type .llcrypt unregistered [ 1876.118592] Key type ._llcrypt unregistered [ 1881.590669] Key type ._llcrypt registered [ 1881.603546] Key type .llcrypt registered [ 1881.797416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1881.823674] alg: No test for adler32 (adler32-zlib) [ 1882.739437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1882.770280] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1889.918663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1900.443466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.519609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1900.554538] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 1901.496800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 1901.731056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1901.977958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1901.985422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1901.996551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1904.356442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1904.374681] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1916.917937] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:41:54 (1790772114) [ 1917.152284] LNet: 109434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.161254] LNetError: 109434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.173266] LNet: Removed LNI 192.168.206.53@tcp [ 1918.239199] LNet: Removed LNI 192.168.206.53@tcp1 [ 1918.243317] LNetError: 109434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1918.257336] LNetError: 109434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1918.294210] LNet: Removed LNI 10.1.2.3@tcp99 [ 1919.037975] Key type .llcrypt unregistered [ 1919.042826] Key type ._llcrypt unregistered [ 1925.440205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1925.446891] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1927.918727] Key type ._llcrypt registered [ 1927.922451] Key type .llcrypt registered [ 1928.133509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1928.154231] alg: No test for adler32 (adler32-zlib) [ 1929.144911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.215238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.242077] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1929.855333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1929.888356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1931.958819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1931.991668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1932.023256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1932.037425] LNetError: 110180:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1932.065500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1932.105428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 1932.552751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1932.611796] LNet: 110200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.629400] LNetError: 110200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1932.638561] LNet: Removed LNI 10.1.2.3@tcp [ 1933.674576] LNet: Removed LNI 192.168.206.53@tcp [ 1934.209569] Key type .llcrypt unregistered [ 1934.218307] Key type ._llcrypt unregistered [ 1946.152762] Key type ._llcrypt registered [ 1946.156543] Key type .llcrypt registered [ 1947.269722] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:42:25 (1790772145) [ 1947.685716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1947.712684] alg: No test for adler32 (adler32-zlib) [ 1948.636321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.692221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.723616] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1949.390371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 1949.471582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg653-client@tcp [ 1955.291392] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:42:33 (1790772153) [ 1955.488682] LNet: 111731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.505736] LNetError: 111731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.527041] LNet: Removed LNI 192.168.206.53@tcp [ 1956.212925] Key type .llcrypt unregistered [ 1956.218527] Key type ._llcrypt unregistered [ 1962.015342] Key type ._llcrypt registered [ 1962.020431] Key type .llcrypt registered [ 1962.176354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.200276] alg: No test for adler32 (adler32-zlib) [ 1963.180276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1963.220971] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1963.251788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1969.963713] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:42:47 (1790772167) [ 1971.524469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1973.005345] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:42:50 (1790772170) [ 1974.424185] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1976.281590] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:42:54 (1790772174) [ 1976.408539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.434316] LNet: 113148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.443970] LNetError: 113148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.465766] LNet: Removed LNI 192.168.206.53@tcp [ 1976.491890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.556209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.584798] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 1977.337864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1977.398843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1983.923526] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:43:01 (1790772181) [ 1984.056883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.092925] LNet: 113859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.116133] LNetError: 113859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.155558] LNet: Removed LNI 192.168.206.53@tcp [ 1984.203838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.283885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1984.339782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1985.143463] Key type .llcrypt unregistered [ 1985.144695] Key type ._llcrypt unregistered [ 1998.144110] Key type ._llcrypt registered [ 1998.147780] Key type .llcrypt registered [ 1999.471622] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:43:17 (1790772197) [ 1999.884026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.908175] alg: No test for adler32 (adler32-zlib) [ 2000.866850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.918223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.958424] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2001.866592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2001.942673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2008.802596] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:43:26 (1790772206) [ 2009.040536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.058981] LNet: 115418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.070397] LNetError: 115418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.106810] LNet: Removed LNI 192.168.206.53@tcp [ 2009.154369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.247341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2009.305972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2018.271160] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:43:35 (1790772215) [ 2018.665637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.729849] LNet: 115998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.792030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2018.925845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2019.623141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2019.655189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2019.689967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2019.979448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2020.003560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2020.060381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.090665] LNet: 116031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.114826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.163283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2020.422459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2020.448984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2020.508395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2020.792283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2020.817940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2030.303972] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:43:47 (1790772227) [ 2030.421749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.505508] LNet: 116627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.618846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.891294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.967152] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2032.945870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2033.005329] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2035.218320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2035.281150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2035.491771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.522323] LNet: 116912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.531472] LNetError: 116912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.539648] LNet: Removed LNI 192.168.206.53@tcp [ 2035.551320] LNet: Removed LNI 192.168.206.53@tcp2 [ 2035.568787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.688671] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2035.705778] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2035.724779] LNet: 116918:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2036.654923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2036.896705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.933724] LNetError: 116955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.936944] LNetError: 116955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2036.955484] LNet: Removed LNI 192.168.206.53@tcp [ 2036.971705] LNet: Removed LNI 192.168.206.53@tcp2 [ 2037.020564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.083686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2037.107799] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2037.671223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2037.696055] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2038.627531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2038.696964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2038.772016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2039.205529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.264460] LNetError: 117239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.270443] LNetError: 117239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2039.279795] LNet: Removed LNI 192.168.206.53@tcp [ 2040.295120] LNet: Removed LNI 192.168.206.53@tcp2 [ 2040.338552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.389736] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2040.403416] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2040.427233] LNet: 117245:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2041.289865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2041.539926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.561826] LNet: Removed LNI 192.168.206.53@tcp [ 2042.598814] LNet: Removed LNI 192.168.206.53@tcp2 [ 2042.654780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.685273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.702582] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2043.731946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2043.748734] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2044.592050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2044.682953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2044.891659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2044.909614] LNet: 117562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.921578] LNet: 117562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2044.927743] LNetError: 117562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.936519] LNetError: 117562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2044.946795] LNet: Removed LNI 192.168.206.53@tcp [ 2044.959546] LNet: Removed LNI 192.168.206.53@tcp2 [ 2045.032770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.171711] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2045.185792] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2045.202528] LNet: 117568:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2045.751834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2046.073901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2047.135777] LNet: Removed LNI 192.168.206.53@tcp [ 2047.151936] LNet: Removed LNI 192.168.206.53@tcp2 [ 2047.195880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.292909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.325989] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2047.916126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2047.932889] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2048.755538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2048.805896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2048.858905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2049.042955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.079171] LNet: Removed LNI 192.168.206.53@tcp [ 2051.109224] LNet: Removed LNI 192.168.206.53@tcp2 [ 2051.154586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.262724] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2051.274798] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2051.279592] LNet: 117896:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2051.876513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2052.143789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.182544] LNet: Removed LNI 192.168.206.53@tcp [ 2053.224249] LNet: Removed LNI 192.168.206.53@tcp2 [ 2053.241873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.315672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.334442] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2053.922911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2053.942539] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2054.851770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2054.899868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2054.963565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2055.314858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2055.330038] LNetError: 118230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.334657] LNetError: 118230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2055.341568] LNet: Removed LNI 192.168.206.53@tcp [ 2055.353056] LNet: Removed LNI 192.168.206.53@tcp2 [ 2055.385616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.452835] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2055.459864] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2055.465820] LNet: 118236:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2056.116870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2056.526957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.570333] LNet: Removed LNI 192.168.206.53@tcp [ 2057.595867] LNet: Removed LNI 192.168.206.53@tcp2 [ 2057.632766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.719417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2057.753845] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2058.658096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2058.710330] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2059.642503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2059.690630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2059.725815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2059.783651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2060.208672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2061.279371] LNet: Removed LNI 192.168.206.53@tcp [ 2062.312831] LNet: Removed LNI 192.168.206.53@tcp2 [ 2062.335387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.409178] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2062.418162] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2063.259581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2071.735909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2073.727577] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:44:31 (1790772271) [ 2073.892151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.916573] LNet: 119353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.920516] LNet: 119353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2073.932075] LNetError: 119353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.937389] LNetError: 119353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2073.944142] LNet: Removed LNI 192.168.206.53@tcp [ 2073.995874] LNet: Removed LNI 192.168.206.53@tcp2 [ 2074.030790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.355399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.394198] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2075.203459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2075.223704] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2076.251335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.288256] LNet: Removed LNI 10.1.2.3@tcp [ 2076.296527] LNet: Removed LNI 192.168.206.53@tcp [ 2076.316489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.404298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2076.443281] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2076.447607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2077.030477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2077.062717] LNet: Removed LNI 192.168.206.53@tcp [ 2077.076774] LNet: Removed LNI 10.1.2.3@tcp [ 2084.153608] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:44:41 (1790772281) [ 2084.339607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.407938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.473684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.497860] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2085.205840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2085.273067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2085.367880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.445331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.519846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.591535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.640965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.697617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.765775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.835051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.887592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2085.941866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2086.084604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2093.443510] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 08:44:50 (1790772290) [ 2093.714208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.821490] LNet: Removed LNI 192.168.206.53@tcp [ 2094.893085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.935904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2094.965827] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2095.875898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2095.918556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2095.954842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.008818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.060209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.116657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.171818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.242349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.286938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.316952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.354814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.394992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.513450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2096.561239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.633580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.712244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.761592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.814540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.855963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.896532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.935981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2096.972687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2097.018740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.53@tcp [ 2103.883271] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:45:01 (1790772301) [ 2104.204597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.254445] LNet: Removed LNI 192.168.206.53@tcp [ 2105.310854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.498205] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2105.782889] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2105.792231] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2105.801129] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2108.209404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.254699] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2109.154415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2109.197782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2111.277549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2111.361290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2111.473583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.53@tcp -m PUT -e local_timeout -r 1 [ 2111.573685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2111.635740] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2111.641346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2111.979821] LNetError: 121788:0:(lib-move.c:4794:lnet_parse()) 192.168.206.53@tcp, src 192.168.206.53@tcp: Dropping PUT (error -108 looking up sender) [ 2112.671589] LNet: 122247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.682626] LNet: 122247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2112.694582] LNetError: 122247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2112.714307] LNetError: 122247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2112.735055] LNet: Removed LNI 10.1.2.3@tcp [ 2113.765676] LNet: Removed LNI 192.168.206.53@tcp [ 2114.176615] Key type .llcrypt unregistered [ 2114.179088] Key type ._llcrypt unregistered [ 2127.369377] Key type ._llcrypt registered [ 2127.373627] Key type .llcrypt registered [ 2128.708862] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:45:26 (1790772326) [ 2129.529947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2129.545471] alg: No test for adler32 (adler32-zlib) [ 2130.469383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.748546] LNet: 123114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.400572] Key type .llcrypt unregistered [ 2131.403656] Key type ._llcrypt unregistered [ 2138.853746] Key type ._llcrypt registered [ 2138.857772] Key type .llcrypt registered [ 2139.175956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2139.207796] alg: No test for adler32 (adler32-zlib) [ 2140.139528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2140.185393] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2149.559123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2149.813492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2150.724512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.153@tcp [ 2151.085376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2151.143943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2151.212194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2151.255635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2151.307920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2151.397188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2151.455316] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 2151.496218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2151.572845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2151.621639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2151.640661] LNet: 123566:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.689891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2151.767764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2151.821538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2151.865424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2151.937841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2152.006669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2152.938510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2153.264798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2153.308397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2153.346358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2153.403643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2153.448190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2153.472186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2153.497851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2153.523520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2153.551494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2153.565746] LNet: 123685:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.589262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2153.629674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2153.670850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2153.707717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2153.772294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2153.823950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2154.502956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.153@tcp [ 2154.737512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.765413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.802653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.834072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.873410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2154.918801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.947438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.984388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.017794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2155.040455] LNet: 123804:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2155.063915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2155.103485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2155.132433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2155.181352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2155.234466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2155.280907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2156.147917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2156.414650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2156.448469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2156.484967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2156.508445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2156.554251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2156.608790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2156.653855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2156.693681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2156.727286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2156.791580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2156.824257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2156.852892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2156.878178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2156.910786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.944745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2157.306872] LNet: 123970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.311418] LNetError: 123970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.343568] LNet: Removed LNI 192.168.206.53@tcp [ 2157.729447] Key type .llcrypt unregistered [ 2157.732136] Key type ._llcrypt unregistered [ 2176.267118] Key type ._llcrypt registered [ 2176.270257] Key type .llcrypt registered [ 2177.461788] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:46:15 (1790772375) [ 2177.944891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2177.956943] alg: No test for adler32 (adler32-zlib) [ 2178.918923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.377486] LNet: 124857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.038515] Key type .llcrypt unregistered [ 2180.043073] Key type ._llcrypt unregistered [ 2186.872099] Key type ._llcrypt registered [ 2186.874027] Key type .llcrypt registered [ 2187.084273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.096666] alg: No test for adler32 (adler32-zlib) [ 2188.042493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2188.084174] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2196.720945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2206.427693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.494486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2206.524064] LNet: Added LNI 192.168.206.53@tcp1 [8/256/0/180] [ 2207.687988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.153@tcp [ 2207.814398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2208.393783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.153@tcp [ 2208.620642] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.153@tcp [ 2208.674818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2208.703748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2208.734880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2208.776631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2208.836240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2208.898786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2208.932649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2208.963103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2209.001060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2209.047782] LNet: 125516:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2209.090168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2209.160141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2209.219943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2209.294043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2209.372759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2209.433210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2210.342778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2210.611351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2210.646104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2210.699738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2210.732310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2210.774737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2210.842776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2210.885796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2210.911174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2210.934989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2210.947962] LNet: 125635:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2210.962404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2210.985224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2211.022728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2211.083062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2211.131210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2211.166571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2211.771576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.153@tcp [ 2211.983920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2212.047607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2212.134415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2212.205781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2212.233551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2212.262079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2212.311286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2212.353059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2212.380779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2212.393302] LNet: 125755:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2212.404709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2212.423618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2212.443687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2212.461962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2212.479446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2212.498354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2213.306822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2213.592318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2213.618414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2213.642942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2213.668888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2213.707453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2213.769345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2213.812621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2213.864938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2213.922182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2214.004912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2214.047294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2214.098661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2214.154472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2214.208438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2214.279862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2220.615694] LNet: 125955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.625860] LNetError: 125955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2220.639994] LNet: Removed LNI 192.168.206.53@tcp [ 2220.664506] LNet: Removed LNI 192.168.206.53@tcp1 [ 2221.332356] Key type .llcrypt unregistered [ 2221.336920] Key type ._llcrypt unregistered [ 2236.740905] Key type ._llcrypt registered [ 2236.747492] Key type .llcrypt registered [ 2237.983184] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:47:15 (1790772435) [ 2238.529970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2238.545547] alg: No test for adler32 (adler32-zlib) [ 2239.469808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.508787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2239.530949] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2240.131819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2240.159108] LNet: Added LNI 192.168.206.53@tcp2 [8/256/0/180] [ 2240.861455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.53@tcp [ 2240.957143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2242.015610] LNet: Removed LNI 192.168.206.53@tcp [ 2248.121656] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:47:25 (1790772445) [ 2249.559202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2251.092499] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:47:29 (1790772449) [ 2259.353722] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:47:36 (1790772456) [ 2267.356842] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:47:44 (1790772464) [ 2275.837583] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:47:53 (1790772473) [ 2286.347219] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:48:03 (1790772483) [ 2293.841946] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:48:11 (1790772491) [ 2300.753275] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:48:18 (1790772498) [ 2309.468523] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:48:27 (1790772507) [ 2317.251827] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:48:34 (1790772514) [ 2324.094417] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:48:41 (1790772521) [ 2330.406462] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:48:48 (1790772528) [ 2338.864390] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:48:56 (1790772536) [ 2347.541493] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:49:05 (1790772545) [ 2355.878397] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:49:13 (1790772553) [ 2363.149579] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:49:20 (1790772560) [ 2371.808261] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:49:29 (1790772569) [ 2382.652980] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:49:39 (1790772579) [ 2392.058558] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:49:49 (1790772589) [ 2392.287972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.334410] LNet: 137666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.346704] LNetError: 137666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2392.386163] LNet: Removed LNI 192.168.206.53@tcp2 [ 2392.450599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.597378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.627705] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2393.938663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.992205] LNet: 137811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2393.999645] LNetError: 137811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2394.017231] LNet: Removed LNI 192.168.206.53@tcp [ 2394.061638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.102331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2394.123003] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2401.745439] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:49:58 (1790772598) [ 2408.569354] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:50:06 (1790772606) [ 2415.336337] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:50:13 (1790772613) [ 2422.544975] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:50:20 (1790772620) [ 2428.745799] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:50:26 (1790772626) [ 2435.895436] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:50:33 (1790772633) [ 2443.183239] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:50:41 (1790772641) [ 2451.331778] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:50:49 (1790772649) [ 2458.494187] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:50:56 (1790772656) [ 2465.960662] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:51:03 (1790772663) [ 2473.471735] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:51:11 (1790772671) [ 2480.314807] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:51:17 (1790772677) [ 2487.493514] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2291 sec ========== 08:51:25 (1790772685) [ 2487.964812] LNet: 145275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2487.971925] LNetError: 145275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.986306] LNet: Removed LNI 192.168.206.53@tcp [ 2488.776273] Key type .llcrypt unregistered [ 2488.779260] Key type ._llcrypt unregistered [ 2502.340770] Key type ._llcrypt registered [ 2502.354301] Key type .llcrypt registered [ 2503.094152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2503.105270] alg: No test for adler32 (adler32-zlib) [ 2504.221693] Lustre: Lustre: Build Version: 2.17.59_3_gf57cca5 [ 2504.534929] LNet: Added LNI 192.168.206.53@tcp [8/256/0/180] [ 2506.223227] Key type lgssc registered [ 2507.518786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.227617] Lustre: Mounted lustre-client - version 2.17.59_3_gf57cca5 [ 2584.938447] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2592.655162] Lustre: Unmounted lustre-client [ 2630.942546] Key type lgssc unregistered [ 2631.311678] LNet: 148239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.318761] LNetError: 148239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2631.330115] LNet: Removed LNI 192.168.206.53@tcp [ 2632.081519] Key type .llcrypt unregistered [ 2632.083607] Key type ._llcrypt unregistered