[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 465149496 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2856376K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002427] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006725] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010060] Yama: becoming mindful. [ 0.011037] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.020490] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025761] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029086] *** VALIDATE tmpfs *** [ 0.030465] *** VALIDATE proc *** [ 0.031260] *** VALIDATE cgroup *** [ 0.032010] *** VALIDATE cgroup2 *** [ 0.033244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036033] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039951] debug: unmapping init [mem 0xffffffff9cc59000-0xffffffff9cc60fff] [ 0.042000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.043940] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.044994] ... value mask: 0000ffffffffffff [ 0.045012] ... max period: 00007fffffffffff [ 0.046012] ... fixed-purpose events: 3 [ 0.046981] ... event mask: 000000070000000f [ 0.047340] rcu: Hierarchical SRCU implementation. [ 0.049534] smp: Bringing up secondary CPUs ... [ 0.050624] x86: Booting SMP configuration: [ 0.051032] .... node #0, CPUs: #1 #2 #3 [ 0.055116] smp: Brought up 1 node, 4 CPUs [ 0.057018] smpboot: Max logical packages: 1 [ 0.058017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.239024] node 0 deferred pages initialised in 177ms [ 0.243207] devtmpfs: initialized [ 0.244276] x86/mm: Memory block size: 128MB [ 0.247000] gcov: version magic: 0x41383552 [ 0.249702] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.257330] pinctrl core: initialized pinctrl subsystem [ 0.258216] [ 0.258792] ************************************************************* [ 0.261017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264014] ** ** [ 0.266014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269018] ** ** [ 0.271012] ** This means that this kernel is built to expose internal ** [ 0.274018] ** IOMMU data structures, which may compromise security on ** [ 0.276016] ** your system. ** [ 0.279013] ** ** [ 0.281015] ** If you see this message and you are not debugging the ** [ 0.284017] ** kernel, report this immediately to your vendor! ** [ 0.286014] ** ** [ 0.288016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291014] ************************************************************* [ 0.292712] NET: Registered protocol family 16 [ 0.294428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.297070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303179] cpuidle: using governor menu [ 0.304952] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.305817] PCI: Using configuration type 1 for base access [ 0.307129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.318095] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.321114] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325201] cryptd: max_cpu_qlen set to 1000 [ 0.327305] ACPI: Added _OSI(Module Device) [ 0.329025] ACPI: Added _OSI(Processor Device) [ 0.331037] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.332017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337480] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343428] ACPI: Interpreter enabled [ 0.345092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.347018] ACPI: Using IOAPIC for interrupt routing [ 0.349194] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.352501] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.362251] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.365047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368034] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371134] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.376329] acpiphp: Slot [2] registered [ 0.378156] acpiphp: Slot [5] registered [ 0.379163] acpiphp: Slot [6] registered [ 0.381135] acpiphp: Slot [3] registered [ 0.382104] acpiphp: Slot [4] registered [ 0.384120] acpiphp: Slot [7] registered [ 0.385107] acpiphp: Slot [8] registered [ 0.387136] acpiphp: Slot [9] registered [ 0.388151] acpiphp: Slot [10] registered [ 0.390175] acpiphp: Slot [11] registered [ 0.391125] acpiphp: Slot [12] registered [ 0.393175] acpiphp: Slot [13] registered [ 0.395156] acpiphp: Slot [14] registered [ 0.396125] acpiphp: Slot [15] registered [ 0.397147] acpiphp: Slot [16] registered [ 0.398080] acpiphp: Slot [17] registered [ 0.400091] acpiphp: Slot [18] registered [ 0.401135] acpiphp: Slot [19] registered [ 0.402137] acpiphp: Slot [20] registered [ 0.403142] acpiphp: Slot [21] registered [ 0.404067] acpiphp: Slot [22] registered [ 0.404879] acpiphp: Slot [23] registered [ 0.406061] acpiphp: Slot [24] registered [ 0.407065] acpiphp: Slot [25] registered [ 0.408104] acpiphp: Slot [26] registered [ 0.409119] acpiphp: Slot [27] registered [ 0.410130] acpiphp: Slot [28] registered [ 0.412121] acpiphp: Slot [29] registered [ 0.413135] acpiphp: Slot [30] registered [ 0.415138] acpiphp: Slot [31] registered [ 0.416089] PCI host bridge to bus 0000:00 [ 0.418032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.421057] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.423034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428039] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.430044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.432226] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.435602] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.438251] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.446000] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.450055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.453022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.455018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.459573] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.462857] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.465057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.468547] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.471921] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.481025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.485021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.491534] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.496902] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.508020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.527025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.535198] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.544020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.551022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.564022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.578492] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.581521] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.583493] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.585310] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.587214] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.592187] iommu: Default domain type: Passthrough [ 0.593554] SCSI subsystem initialized [ 0.595192] ACPI: bus type USB registered [ 0.597142] usbcore: registered new interface driver usbfs [ 0.599098] usbcore: registered new interface driver hub [ 0.601147] usbcore: registered new device driver usb [ 0.603241] pps_core: LinuxPPS API ver. 1 registered [ 0.605028] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.607089] PTP clock support registered [ 0.609039] EDAC MC: Ver: 3.0.0 [ 0.610456] PCI: Using ACPI for IRQ routing [ 0.612769] NetLabel: Initializing [ 0.614010] NetLabel: domain hash size = 128 [ 0.615013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.617092] NetLabel: unlabeled traffic allowed by default [ 0.619318] vgaarb: loaded [ 0.621309] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.623018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.631124] clocksource: Switched to clocksource kvm-clock [ 0.742610] VFS: Disk quotas dquot_6.6.0 [ 0.743950] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.746468] *** VALIDATE ramfs *** [ 0.747747] *** VALIDATE hugetlbfs *** [ 0.749255] pnp: PnP ACPI init [ 0.751695] pnp: PnP ACPI: found 6 devices [ 0.770279] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.773533] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.775925] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.777973] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.780032] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.782556] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.785196] NET: Registered protocol family 2 [ 0.787568] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.792724] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.797022] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802507] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.805940] TCP: Hash tables configured (established 65536 bind 65536) [ 0.809025] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.812141] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.815113] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.817875] NET: Registered protocol family 1 [ 0.821315] RPC: Registered named UNIX socket transport module. [ 0.823655] RPC: Registered udp transport module. [ 0.826092] RPC: Registered tcp transport module. [ 0.827778] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.830170] NET: Registered protocol family 44 [ 0.831710] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.833990] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.835742] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.838189] PCI: CLS 0 bytes, default 64 [ 0.840234] Unpacking initramfs... [ 2.264927] debug: unmapping init [mem 0xffff9e3ffcc64000-0xffff9e3ffffcffff] [ 2.268789] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.272159] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.274857] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.779075] Initialise system trusted keyrings [ 2.780938] Key type blacklist registered [ 2.782951] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.793867] zbud: loaded [ 2.797286] *** VALIDATE nfs *** [ 2.798473] *** VALIDATE nfs4 *** [ 2.799929] pstore: using deflate compression [ 2.809463] Platform Keyring initialized [ 2.935972] NET: Registered protocol family 38 [ 2.937761] Key type asymmetric registered [ 2.939374] Asymmetric key parser 'x509' registered [ 2.941320] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.944405] io scheduler mq-deadline registered [ 2.946339] io scheduler kyber registered [ 2.948317] io scheduler bfq registered [ 2.950431] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.953801] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.956782] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.959889] ACPI: Power Button [PWRF] [ 2.965475] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.971676] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.984731] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.022349] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.059563] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.064763] Non-volatile memory driver v1.3 [ 3.066512] Linux agpgart interface v0.103 [ 3.110494] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.113470] vda: detected capacity change from 0 to 75083776 [ 3.134562] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.137456] vdb: detected capacity change from 0 to 1073741824 [ 3.146910] libphy: Fixed MDIO Bus: probed [ 3.152369] usbcore: registered new interface driver usbserial_generic [ 3.154675] usbserial: USB Serial support registered for generic [ 3.156932] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.161144] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.163185] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.165315] mousedev: PS/2 mouse device common for all mice [ 3.168383] rtc_cmos 00:05: RTC can wake from S4 [ 3.171775] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.175712] rtc_cmos 00:05: registered as rtc0 [ 3.178233] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.178552] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.184617] intel_pstate: CPU model not supported [ 3.184707] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.190562] hid: raw HID events driver (C) Jiri Kosina [ 3.192640] usbcore: registered new interface driver usbhid [ 3.194935] usbhid: USB HID core driver [ 3.196759] drop_monitor: Initializing network drop monitor service [ 3.199052] Initializing XFRM netlink socket [ 3.200893] NET: Registered protocol family 10 [ 3.203686] Segment Routing with IPv6 [ 3.205052] NET: Registered protocol family 17 [ 3.207077] mpls_gso: MPLS GSO support [ 3.213493] RAS: Correctable Errors collector initialized. [ 3.215621] AVX version of gcm_enc/dec engaged. [ 3.217104] AES CTR mode by8 optimization enabled [ 3.300478] sched_clock: Marking stable (3300454944, 0)->(4226838741, -926383797) [ 3.303750] registered taskstats version 1 [ 3.305663] Loading compiled-in X.509 certificates [ 3.308325] zswap: loaded using pool lzo/zbud [ 3.338173] Key type big_key registered [ 3.353162] Key type encrypted registered [ 3.354790] ima: No TPM chip found, activating TPM-bypass! [ 3.356789] ima: Allocated hash algorithm: sha1 [ 3.358521] ima: No architecture policies found [ 3.359814] evm: Initialising EVM extended attributes: [ 3.361626] evm: security.selinux [ 3.362927] evm: security.ima [ 3.364183] evm: security.capability [ 3.365508] evm: HMAC attrs: 0x1 [ 3.368111] rtc_cmos 00:05: setting system clock to 2026-09-03 15:07:59 UTC (1788448079) [ 3.374542] debug: unmapping init [mem 0xffffffff9dc03000-0xffffffff9ddfffff] [ 3.377744] debug: unmapping init [mem 0xffffffff9c982000-0xffffffff9cc58fff] [ 3.384150] Write protecting the kernel read-only data: 28672k [ 3.387665] debug: unmapping init [mem 0xffffffff9b003000-0xffffffff9b1fffff] [ 3.390677] debug: unmapping init [mem 0xffffffff9b914000-0xffffffff9b9fffff] [ 3.422316] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.430950] systemd[1]: Detected virtualization kvm. [ 3.432980] systemd[1]: Detected architecture x86-64. [ 3.434945] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.459044] systemd[1]: No hostname configured. [ 3.460928] systemd[1]: Set hostname to . [ 3.463075] random: systemd: uninitialized urandom read (16 bytes read) [ 3.465670] systemd[1]: Initializing machine ID from random generator. [ 3.618747] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621401] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.626407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629199] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.636284] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.299495] device-mapper: uevent: version 1.0.3 [ 4.301884] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.998565] virtio_net virtio0 ens2: renamed from eth0 [ 5.018085] random: fast init done [ 5.044164] scsi host0: ata_piix [ 5.071001] scsi host1: ata_piix [ 5.072620] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.075148] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.497560] dracut-initqueue[577]: RTNETLINK answers: File exists [ 9.878572] random: crng init done [ 9.882254] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.528277] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.310979] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.978241] SELinux: Disabled at runtime. [ 14.087965] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.112460] systemd[1]: Detected virtualization kvm. [ 14.116411] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.529864] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.542378] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.549175] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.555980] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.564421] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.581129] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.597824] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ 15.846880] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.706735] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.537185] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.587307] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.814868] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.865774] EDAC sbridge: Ver: 1.1.2 [ 20.475059] Key type dns_resolver registered [ 20.877830] NFS: Registering the id_resolver key type [ 20.880191] Key type id_resolver registered [ 20.881759] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-client login: [ 51.751338] hrtimer: interrupt took 7159418 ns [ 90.175145] libcfs: loading out-of-tree module taints kernel. [ 90.567239] Key type ._llcrypt registered [ 90.577991] Key type .llcrypt registered [ 92.161362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 92.211900] alg: No test for adler32 (adler32-zlib) [ 94.282661] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 95.443022] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 97.328448] Key type lgssc registered [ 100.859141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 238.279750] Lustre: Mounted lustre-client - version 2.17.57_103_g02cdcd5 [ 244.499664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.532495] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing check_logdir /tmp/testlogs/ [ 263.955337] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing yml_node [ 264.159163] Lustre: lustre-OST0000-osc-ffff9e404a75f800: disconnect after 23s idle [ 268.338142] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 271.881465] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 274.873041] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 276.769052] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 3 11:12:31 EDT 2026 [ 292.646734] Lustre: Unmounted lustre-client [ 319.407263] Key type lgssc unregistered [ 319.669770] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.678879] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.700200] LNet: Removed LNI 192.168.204.42@tcp [ 320.496187] Key type .llcrypt unregistered [ 320.499839] Key type ._llcrypt unregistered [ 350.616750] Key type ._llcrypt registered [ 350.618599] Key type .llcrypt registered [ 350.863185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 350.880797] alg: No test for adler32 (adler32-zlib) [ 351.871774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 351.965685] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 351.994843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 355.707462] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 355.952760] LNet: 9300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.961118] LNetError: 9300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.982194] LNet: Removed LNI 192.168.204.42@tcp [ 356.634289] Key type .llcrypt unregistered [ 356.638231] Key type ._llcrypt unregistered [ 362.688972] Key type ._llcrypt registered [ 362.696546] Key type .llcrypt registered [ 364.042049] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:13:58 (1788448438) [ 364.497713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 364.514896] alg: No test for adler32 (adler32-zlib) [ 365.413991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.493559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.515957] LNet: 9708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.539401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.600483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 365.623523] LNet: 9715:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 374.136867] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:14:08 (1788448448) [ 374.226123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.253207] LNet: 10285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.282722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.430380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 381.042590] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:14:15 (1788448455) [ 381.165963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.197908] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.225331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.340577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 388.339478] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:14:23 (1788448463) [ 388.567272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.584556] LNet: 11461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.607387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.698164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 395.102546] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:14:29 (1788448469) [ 395.220281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.240478] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.287803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.381659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 395.915958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 402.895569] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:14:37 (1788448477) [ 403.249284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.271802] LNet: 12638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.300316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.388261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.851367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 404.286754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.734470] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:14:45 (1788448485) [ 410.837133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.874490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.960392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 417.537946] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:14:51 (1788448491) [ 417.672190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.718477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.803716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 417.846236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.246151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 418.317043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.003477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 419.102550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.909928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 419.981943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 420.582595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 420.637934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.209894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 421.245213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.860327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 421.896979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.301295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 422.368585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.763265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.794739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 423.380055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 423.438371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.872830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.916922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.505962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.576583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.999507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 425.022122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.150055] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:15:06 (1788448506) [ 432.457754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.479266] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.482900] LNet: 14501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 432.502426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.608565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 432.661767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 441.521571] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:15:15 (1788448515) [ 441.746988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.851981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.098851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 442.149426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 450.128864] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:15:24 (1788448524) [ 450.282248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.382893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.501532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 450.588760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 460.867198] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:15:35 (1788448535) [ 460.927983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.024614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.181922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 461.248621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 471.135376] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:15:45 (1788448545) [ 471.277959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.322593] LNet: 16842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.328499] LNet: 16842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 471.362722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.506735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 471.553362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 479.410724] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:15:54 (1788448554) [ 479.685178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.725012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.817473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 479.846202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.573479] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:16:01 (1788448561) [ 487.794325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.870927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.021300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.629786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 495.512647] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:16:10 (1788448570) [ 495.830401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.893139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.244375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.485951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.252049] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:16:18 (1788448578) [ 504.570768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.641261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.810561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 505.080695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 513.827747] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:16:27 (1788448587) [ 513.968847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.039804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.947574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.415365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.829396] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:16:38 (1788448598) [ 524.291814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.351766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.422751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.472371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 531.499424] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:16:45 (1788448605) [ 531.674447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.736574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.898826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 531.997932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.249438] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:16:54 (1788448614) [ 540.401469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.460161] LNet: 21704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.480953] LNet: 21704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 540.583194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.720587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 547.820604] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:17:02 (1788448622) [ 547.971084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.043793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.195537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 556.835379] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:17:10 (1788448630) [ 556.949578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.012206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.147903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 557.945533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 565.815898] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:17:20 (1788448640) [ 565.967192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.018656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.133223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 566.182589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 573.941977] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:17:28 (1788448648) [ 574.024948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.076256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.224809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 574.277848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 581.069527] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:17:35 (1788448655) [ 581.208601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.275273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.358853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 581.403117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 588.320816] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:17:43 (1788448663) [ 588.418932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.514258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.649505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 588.693288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 596.063908] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:17:50 (1788448670) [ 596.182780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.278962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.380312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 596.517910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 596.546399] LNetError: 25822:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 596.583561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 596.620122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 603.893676] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:17:58 (1788448678) [ 604.017850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.060097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.201908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AeuqsiMuIV [ 604.289836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AeuqsiMuIV [ 604.360767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.AeuqsiMuIV [ 612.350734] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:18:06 (1788448686) [ 612.462054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.538980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.664654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 612.732621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 612.833932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 622.376422] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 11:18:16 (1788448696) [ 622.533173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.643550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.816913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.879240] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 625.062872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 626.357898] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 628.295193] Key type lgssc registered [ 629.616165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 633.391872] LNetError: 27584:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.42@tcp99 has no available nets [ 636.401231] Key type lgssc unregistered [ 636.844748] LNetError: 28693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.862485] LNet: Removed LNI 192.168.204.42@tcp [ 637.566295] Key type .llcrypt unregistered [ 637.569570] Key type ._llcrypt unregistered [ 644.964785] Key type ._llcrypt registered [ 644.971324] Key type .llcrypt registered [ 646.233865] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:18:40 (1788448720) [ 646.843440] Key type .llcrypt unregistered [ 646.847334] Key type ._llcrypt unregistered [ 652.328983] Key type ._llcrypt registered [ 652.331984] Key type .llcrypt registered [ 652.443140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 652.483872] alg: No test for adler32 (adler32-zlib) [ 653.484438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.573501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 653.618130] LNet: 29602:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 653.625224] LNet: 29602:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 653.634941] LNet: 29602:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 653.660284] LNet: Message forwarding will be enabled in 2s [ 656.735253] LNet: Message forwarding enabled [ 657.159675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 657.184942] LNet: Message forwarding will stop in 2s to 6s [ 660.838457] LNet: Message forwarding disabled [ 661.901597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 661.932533] LNet: 29700:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 661.940626] LNet: 29700:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 661.949161] LNet: 29700:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 661.970467] LNet: Message forwarding will be enabled in 2s [ 662.278365] LNet: 29713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.087692] Key type .llcrypt unregistered [ 663.093630] Key type ._llcrypt unregistered [ 674.600812] Key type ._llcrypt registered [ 674.609480] Key type .llcrypt registered [ 676.174054] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:19:10 (1788448750) [ 676.583967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.603577] alg: No test for adler32 (adler32-zlib) [ 677.523915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.582822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 684.288391] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:19:18 (1788448758) [ 684.416020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.452915] LNet: 31108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.497859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.580433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 684.649933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 684.727652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 684.781390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 684.840196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 684.882970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 684.915779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 684.970341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 685.023193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 685.067922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 685.111171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 685.171210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 685.231918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 685.273726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 685.309364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 685.373374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 685.453572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 685.565626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 685.623479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 685.660921] LNetError: 31157:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 694.016918] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:19:28 (1788448768) [ 694.173342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.249562] LNet: 31723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.294489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.454536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.679754] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:19:36 (1788448776) [ 704.652241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg442-server [ 706.734515] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:19:41 (1788448781) [ 706.796412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.821633] LNet: 32653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.839235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.927847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.962144] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 707.849675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[43-47]@tcp [ 707.880815] LNet: 32794:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 707.949091] LNetError: 32660:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 707.978424] LNetError: 32660:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.44@tcp has gone from up to down [ 715.047632] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:19:49 (1788448789) [ 717.090665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg442-server [ 718.845406] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:19:53 (1788448793) [ 719.038510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.075335] LNet: 33731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.089349] LNetError: 33731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.113218] LNet: Removed LNI 192.168.204.42@tcp [ 719.148568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.215672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.242311] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 719.885538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 719.903322] LNet: 33848:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 719.912811] LNet: 33848:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 719.936114] LNetError: 33735:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 719.949047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[43-48/2]@tcp [ 719.950799] LNetError: 33735:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 719.950822] LNetError: 33735:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.45@tcp has gone from up to down [ 719.950827] LNetError: 33735:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 727.091254] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:20:01 (1788448801) [ 727.177843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.208436] LNet: 34433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.216618] LNetError: 34433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.238876] LNet: Removed LNI 192.168.204.42@tcp [ 727.263278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.378703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 727.511456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.606564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.702591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.773955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.834154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.891988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 727.953785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.988535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.056955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.140119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.176134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.220921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 728.324936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.361853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.424839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.521882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.569721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.634369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 728.721820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.783433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.933227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.046786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.115691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.164817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 729.245254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.292198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.359027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.470287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.513742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.575230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 729.668475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.704536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.759068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.287527] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:20:11 (1788448811) [ 738.438392] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg442-server [ 740.315786] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:20:15 (1788448815) [ 742.816942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg442-server [ 744.528979] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:20:19 (1788448819) [ 744.705921] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.719127] LNet: 35838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 745.426107] Key type .llcrypt unregistered [ 745.428794] Key type ._llcrypt unregistered [ 751.888286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.975857] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.979578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.993862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.615772] Key type ._llcrypt registered [ 754.618842] Key type .llcrypt registered [ 754.802289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.820640] alg: No test for adler32 (adler32-zlib) [ 755.795318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.849760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.869951] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 756.488196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 756.507859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 758.348209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 758.387380] LNet: Removed LNI 10.1.2.3@tcp [ 758.513670] LNet: 36557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.519904] LNetError: 36557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.533191] LNet: Removed LNI 192.168.204.42@tcp [ 759.200657] Key type .llcrypt unregistered [ 759.203279] Key type ._llcrypt unregistered [ 769.715213] Key type ._llcrypt registered [ 769.720778] Key type .llcrypt registered [ 770.990577] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:20:45 (1788448845) [ 771.566692] Key type .llcrypt unregistered [ 771.570911] Key type ._llcrypt unregistered [ 777.725932] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.837308] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.852249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.860566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 780.527776] Key type ._llcrypt registered [ 780.530515] Key type .llcrypt registered [ 780.697470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.718560] alg: No test for adler32 (adler32-zlib) [ 781.612771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.679781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.714425] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 782.585915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 782.609142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.631893] LNet: 38045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.639449] LNetError: 38045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 785.645831] LNet: Removed LNI 10.1.2.3@tcp [ 785.682655] LNet: Removed LNI 192.168.204.42@tcp [ 786.514539] Key type .llcrypt unregistered [ 786.522893] Key type ._llcrypt unregistered [ 800.246593] Key type ._llcrypt registered [ 800.248803] Key type .llcrypt registered [ 801.516259] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:21:16 (1788448876) [ 802.306390] Key type .llcrypt unregistered [ 802.310632] Key type ._llcrypt unregistered [ 808.876829] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 809.073529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 809.089165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 809.100869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.034159] Key type ._llcrypt registered [ 812.037619] Key type .llcrypt registered [ 812.326243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.351978] alg: No test for adler32 (adler32-zlib) [ 813.356126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.406621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 813.428342] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 815.289926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 815.339798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.560806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 817.585429] LNet: Removed LNI 10.1.2.3@tcp [ 817.614407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 817.715642] LNet: Removed LNI 10.1.2.31@tcp [ 817.752902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 817.798295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.810900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 818.830847] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 820.300567] LNet: 39961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.314477] LNetError: 39961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 820.326762] LNet: Removed LNI 10.1.2.31@tcp [ 820.337887] LNet: Removed LNI 10.1.2.3@tcp [ 821.248142] Key type .llcrypt unregistered [ 821.257179] Key type ._llcrypt unregistered [ 839.786212] Key type ._llcrypt registered [ 839.789917] Key type .llcrypt registered [ 841.854039] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:21:55 (1788448915) [ 842.573740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.634179] alg: No test for adler32 (adler32-zlib) [ 843.785509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.909345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.947877] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 845.260520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.301516] LNet: 40912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.345614] LNetError: 40912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.368078] LNet: Removed LNI 192.168.204.42@tcp [ 845.425261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.523848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.42@tcp [ 845.569529] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 847.079609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.42@tcp [ 847.131666] LNet: Removed LNI 192.168.204.42@tcp [ 847.233499] LNet: 41047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.055436] Key type .llcrypt unregistered [ 848.060305] Key type ._llcrypt unregistered [ 857.946592] Key type ._llcrypt registered [ 857.948922] Key type .llcrypt registered [ 859.058449] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:22:13 (1788448933) [ 861.005280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg442-server [ 862.889919] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:22:17 (1788448937) [ 863.608772] Key type .llcrypt unregistered [ 863.614658] Key type ._llcrypt unregistered [ 871.677379] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 871.904068] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 871.918069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 871.939234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 874.571816] Key type ._llcrypt registered [ 874.575476] Key type .llcrypt registered [ 874.869514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 874.894630] alg: No test for adler32 (adler32-zlib) [ 875.871989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.940133] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 877.245624] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 880.138689] LNet: 42879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.144935] LNetError: 42879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.151571] LNet: Removed LNI 192.168.204.42@tcp1 [ 880.167908] LNet: Removed LNI 10.1.2.3@tcp2 [ 881.016812] Key type .llcrypt unregistered [ 881.027355] Key type ._llcrypt unregistered [ 892.758155] Key type ._llcrypt registered [ 892.759389] Key type .llcrypt registered [ 894.414939] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:22:48 (1788448968) [ 894.846707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.868153] alg: No test for adler32 (adler32-zlib) [ 895.850660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.943882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.982153] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 896.563442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.581748] LNet: 43856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.586614] LNetError: 43856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.597849] LNet: Removed LNI 192.168.204.42@tcp [ 896.614948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.679431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 896.764126] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 904.112645] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:22:58 (1788448978) [ 904.442238] LNet: 44449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.450486] LNetError: 44449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.481754] LNet: Removed LNI 192.168.204.42@tcp [ 905.091895] Key type .llcrypt unregistered [ 905.096552] Key type ._llcrypt unregistered [ 911.204692] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.321839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.330361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.334761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.901704] Key type ._llcrypt registered [ 913.904765] Key type .llcrypt registered [ 914.086758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.102460] alg: No test for adler32 (adler32-zlib) [ 915.005688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.061820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.090657] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 915.902829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 915.938635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.728703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.773275] LNet: 45188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.784105] LNetError: 45188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.795892] LNet: Removed LNI 10.1.2.3@tcp [ 918.810200] LNet: Removed LNI 192.168.204.42@tcp [ 918.840974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.908779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 918.934520] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 918.942888] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.789438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.818931] LNet: 45215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.833094] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.846039] LNetError: 45215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.863993] LNet: Removed LNI 10.1.2.3@tcp [ 919.897293] LNet: Removed LNI 192.168.204.42@tcp [ 919.942767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.013442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 920.037412] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 920.040395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.602486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 920.639291] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 921.786407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.822646] LNet: 45378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.839811] LNetError: 45378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.850269] LNetError: 45378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 921.865280] LNet: Removed LNI 10.1.2.3@tcp [ 921.866951] LNet: Removed LNI 192.168.204.42@tcp [ 921.880924] LNet: Removed LNI 192.168.204.42@tcp2 [ 921.912157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.999913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 922.038080] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 922.052853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.073964] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 929.394770] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:23:24 (1788449004) [ 929.565023] LNet: 45971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.585520] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.591920] LNetError: 45971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 929.603835] LNet: Removed LNI 10.1.2.3@tcp [ 929.614055] LNet: Removed LNI 192.168.204.42@tcp [ 929.643214] LNet: Removed LNI 192.168.204.42@tcp2 [ 930.218751] Key type .llcrypt unregistered [ 930.227080] Key type ._llcrypt unregistered [ 936.835895] Key type ._llcrypt registered [ 936.839467] Key type .llcrypt registered [ 937.100861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.128595] alg: No test for adler32 (adler32-zlib) [ 938.040506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.123181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.160159] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 939.086438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.103909] LNet: 46366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.109895] LNetError: 46366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.130959] LNet: Removed LNI 192.168.204.42@tcp [ 939.170155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.248203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.297510] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 939.978181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.020350] LNet: 46396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.044926] LNetError: 46396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.066836] LNet: Removed LNI 192.168.204.42@tcp [ 940.097291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.178672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 940.213688] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 940.646372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.679370] LNet: Removed LNI 192.168.204.42@tcp [ 940.715915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.797558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 940.832412] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 941.519375] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 941.643151] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 941.654905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 941.660770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 944.051663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.101504] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.109497] LNet: 46501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 944.117641] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.124798] LNetError: 46501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 944.143553] LNet: Removed LNI 192.168.204.42@tcp [ 944.177859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.268790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 944.333048] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 952.054258] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:23:46 (1788449026) [ 952.284758] LNet: 47098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.292952] LNetError: 47098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.319886] LNet: Removed LNI 192.168.204.42@tcp [ 953.084797] Key type .llcrypt unregistered [ 953.092844] Key type ._llcrypt unregistered [ 959.812387] Key type ._llcrypt registered [ 959.817468] Key type .llcrypt registered [ 960.067792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.097401] alg: No test for adler32 (adler32-zlib) [ 961.066059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.139595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 961.179199] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 962.470680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.497479] LNet: 47493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.502741] LNetError: 47493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.527199] LNet: Removed LNI 192.168.204.42@tcp [ 962.570783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.652491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 962.695114] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 971.063610] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:24:05 (1788449045) [ 971.173968] LNet: 48080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.184995] LNetError: 48080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.224980] LNet: Removed LNI 192.168.204.42@tcp [ 972.152774] Key type .llcrypt unregistered [ 972.155542] Key type ._llcrypt unregistered [ 978.481993] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 978.628211] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 978.634448] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 978.639731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.317596] Key type ._llcrypt registered [ 981.320535] Key type .llcrypt registered [ 981.468833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.481511] alg: No test for adler32 (adler32-zlib) [ 982.391781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.478503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.531907] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 983.577665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.609098] LNet: 48534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.618211] LNetError: 48534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.645456] LNet: Removed LNI 192.168.204.42@tcp [ 983.692660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.816643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 983.884091] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 990.419866] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:24:25 (1788449065) [ 990.559232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.592758] LNet: 49132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.600263] LNetError: 49132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.617585] LNet: Removed LNI 192.168.204.42@tcp [ 990.646911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.749877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 997.693793] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:24:32 (1788449072) [ 997.819915] LNet: 49705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.630659] Key type .llcrypt unregistered [ 998.632395] Key type ._llcrypt unregistered [ 1005.754471] Key type ._llcrypt registered [ 1005.758081] Key type .llcrypt registered [ 1006.013434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.044945] alg: No test for adler32 (adler32-zlib) [ 1007.062812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.129771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1007.170105] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1008.245835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.266386] LNet: 50109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.276453] LNetError: 50109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.289043] LNet: Removed LNI 192.168.204.42@tcp [ 1008.315608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.364952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1008.384773] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1017.206502] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:24:51 (1788449091) [ 1017.541546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.556976] LNet: 50702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.561616] LNetError: 50702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.580314] LNet: Removed LNI 192.168.204.42@tcp [ 1017.629392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.749620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1025.890644] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:24:59 (1788449099) [ 1027.594053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1029.639641] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:25:04 (1788449104) [ 1031.754290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1033.894394] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:25:08 (1788449108) [ 1035.526571] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1037.529297] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:25:12 (1788449112) [ 1039.301812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1041.328219] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:25:15 (1788449115) [ 1042.820750] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1044.727412] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 11:25:19 (1788449119) [ 1044.995370] LNet: 52757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.848673] Key type .llcrypt unregistered [ 1045.850387] Key type ._llcrypt unregistered [ 1052.463957] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1052.689852] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1052.697318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1052.711763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1055.263735] Key type ._llcrypt registered [ 1055.269100] Key type .llcrypt registered [ 1055.500369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1055.510383] alg: No test for adler32 (adler32-zlib) [ 1056.474876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.559531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1056.579900] LNet: Added LNI 192.168.204.42@tcp [8/512/0/180] [ 1056.585712] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1056.590148] LNet: 53077:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1056.706830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1056.724706] LNet: 53096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1056.731615] LNetError: 53096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.738033] LNet: Removed LNI 10.1.2.3@tcp [ 1056.742881] LNetError: 53096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1056.755613] LNet: Removed LNI 192.168.204.42@tcp [ 1056.804559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.881468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1056.929248] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1057.018874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.052689] LNet: 53120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.059343] LNetError: 53120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.070403] LNet: Removed LNI 192.168.204.42@tcp [ 1057.099675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.209869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1057.233448] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1057.237891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1057.239556] LNet: 53128:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1057.325107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.346893] LNet: 53146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.355575] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1057.371988] LNet: Removed LNI 10.1.2.3@tcp [ 1057.378274] LNetError: 53146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.406755] LNet: Removed LNI 192.168.204.42@tcp [ 1057.443946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.521993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1057.538613] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1057.542934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1057.544482] LNet: 53154:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1057.667305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.695780] LNet: 53180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.704244] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1057.710791] LNet: Removed LNI 10.1.2.3@tcp [ 1057.713748] LNetError: 53180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.736416] LNet: Removed LNI 192.168.204.42@tcp [ 1057.769942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.847448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1057.883412] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1057.902962] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1058.007961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.046436] LNet: 53208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.055505] LNetError: 53208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.060839] LNet: Removed LNI 10.1.2.3@tcp [ 1058.064691] LNetError: 53208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.085497] LNet: Removed LNI 192.168.204.42@tcp [ 1058.107429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.192663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.212187] LNet: Added LNI 192.168.204.42@tcp [8/256/0/182] [ 1058.221171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1058.223917] LNet: 53216:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1058.301958] LNet: Removed LNI 10.1.2.3@tcp [ 1058.318660] LNet: Removed LNI 192.168.204.42@tcp [ 1059.076883] Key type .llcrypt unregistered [ 1059.081477] Key type ._llcrypt unregistered [ 1071.528940] Key type ._llcrypt registered [ 1071.534787] Key type .llcrypt registered [ 1073.328403] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 11:25:47 (1788449147) [ 1075.182769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1077.050394] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:25:51 (1788449151) [ 1077.414820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.451884] alg: No test for adler32 (adler32-zlib) [ 1078.401664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.455249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1078.474188] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1079.196976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.42 [ 1079.216747] LNet: 54478:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1079.222677] LNet: Removed LNI 192.168.204.42@tcp [ 1086.033835] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:26:00 (1788449160) [ 1086.299663] LNet: 55049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.305536] LNetError: 55049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.329884] LNet: Removed LNI 192.168.204.42@tcp [ 1086.920560] Key type .llcrypt unregistered [ 1086.923384] Key type ._llcrypt unregistered [ 1094.004905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1094.252552] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1097.051076] Key type ._llcrypt registered [ 1097.053553] Key type .llcrypt registered [ 1097.356259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.398418] alg: No test for adler32 (adler32-zlib) [ 1099.008905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1107.653399] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:26:21 (1788449181) [ 1107.749930] LNet: 55952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.758710] LNetError: 55952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.778265] LNet: Removed LNI 10.1.2.3@tcp [ 1108.437376] Key type .llcrypt unregistered [ 1108.440477] Key type ._llcrypt unregistered [ 1115.350744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1115.615142] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.624853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.629393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1118.425543] Key type ._llcrypt registered [ 1118.428345] Key type .llcrypt registered [ 1118.595695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.610766] alg: No test for adler32 (adler32-zlib) [ 1119.630220] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.016303] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:26:41 (1788449201) [ 1129.039404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1130.880615] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 11:26:45 (1788449205) [ 1131.281564] LNet: 57141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.290726] LNetError: 57141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1131.306863] LNet: Removed LNI 10.1.2.3@tcp [ 1132.111991] Key type .llcrypt unregistered [ 1132.114211] Key type ._llcrypt unregistered [ 1138.419460] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1138.633141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1138.641985] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1138.649549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1141.393424] Key type ._llcrypt registered [ 1141.394974] Key type .llcrypt registered [ 1141.602322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.614782] alg: No test for adler32 (adler32-zlib) [ 1142.548368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.613880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.100440] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 11:27:05 (1788449225) [ 1151.715471] LNet: 58368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.722691] LNetError: 58368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.740330] LNet: Removed LNI 10.1.2.3@tcp [ 1152.475670] Key type .llcrypt unregistered [ 1152.479695] Key type ._llcrypt unregistered [ 1159.650130] Key type ._llcrypt registered [ 1159.658632] Key type .llcrypt registered [ 1159.964993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.993918] alg: No test for adler32 (adler32-zlib) [ 1160.977121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.024807] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1169.474520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1170.212630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1170.265680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1170.310022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1170.333415] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1170.527464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.563453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.590299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.612755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.637067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.665387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.694872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.717912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.732425] LNet: 58802:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.751987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.783906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.833079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.882217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.931303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.963212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.993159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.519727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1171.561869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1171.601895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1171.622803] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1171.823203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.849943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.880739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.909880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.934922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.960839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.987042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.023772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.051753] LNet: 58912:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.074888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.130801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.184920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.227630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.284043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.319929] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1172.342802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.387354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.872633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1172.902903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1172.938703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1172.955799] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1172.960538] LNet: Skipped 1 previous similar message [ 1173.162016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.189985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.221149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.249949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.274937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.301557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.333405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.359284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.377790] LNet: 59021:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.404255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.445949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.484973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.515197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.555875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.619336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.681105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.116276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1174.147291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1174.190128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1174.438650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.468052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.495180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.535243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.580823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.617760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.654735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.683630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.709427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.750300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.777885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.815570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.847222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.890327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.930188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.505572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1175.554222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1175.593933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1175.611390] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1175.616190] LNet: Skipped 2 previous similar messages [ 1175.795237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.821791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.853729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.885449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.914883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.946839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.982677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.039535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.075277] LNet: 59239:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.084890] LNet: 59239:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1176.117722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.151913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.182766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.220263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.258057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.291728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.332071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.794484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1176.840955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1176.884724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1177.103773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.130916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.154569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.175382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.199630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.226289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.251692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.273269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.299472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.323850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.347581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.373704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.397290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.425749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.450987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.727780] LNet: 59389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.735494] LNetError: 59389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.756770] LNet: Removed LNI 192.168.204.42@tcp [ 1178.192988] Key type .llcrypt unregistered [ 1178.196085] Key type ._llcrypt unregistered [ 1194.179792] Key type ._llcrypt registered [ 1194.181425] Key type .llcrypt registered [ 1195.954310] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 11:27:50 (1788449270) [ 1197.160268] Key type .llcrypt unregistered [ 1197.162414] Key type ._llcrypt unregistered [ 1205.108073] Key type ._llcrypt registered [ 1205.119265] Key type .llcrypt registered [ 1205.253774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.266986] alg: No test for adler32 (adler32-zlib) [ 1206.123445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1206.158830] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1216.580442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1228.414447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.455973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1228.478465] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1229.967586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1230.012326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1230.058939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1230.099585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1230.142779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 1 [ 1230.183290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 1 [ 1230.255507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 1 [ 1230.313364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 1 [ 1230.409663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1230.448511] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1230.458101] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1230.735467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.796548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.850934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.893440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.950673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.986679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.052892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.099054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.121804] LNet: 60852:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.149227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.194886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.263275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.309042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.336688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.373775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.408817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.058279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1232.087568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1232.117931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1232.149216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1232.181737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 1 [ 1232.211442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 1 [ 1232.239814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 1 [ 1232.276936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 1 [ 1232.333582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1232.370813] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1232.380868] LNet: Skipped 2 previous similar messages [ 1232.617382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.647564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.687672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.733548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.775360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.814283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.853770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.898731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.930633] LNet: 60973:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.962541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.997052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.032721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.075884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.104291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.134777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.175447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.811738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1233.862267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1233.907531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1233.962453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1234.008412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 1 [ 1234.037071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 1 [ 1234.066774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 1 [ 1234.095841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 1 [ 1234.133465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1234.148847] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1234.154501] LNet: Skipped 2 previous similar messages [ 1234.342463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.369088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.397371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.425458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.449885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.475597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.498545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.523470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.540906] LNet: 61095:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.560389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.582697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.610159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.633986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.677920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.717779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.755791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1235.265453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1235.296541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1235.327716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1235.369196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1235.402405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 1 [ 1235.433617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 1 [ 1235.505665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 1 [ 1235.551180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 1 [ 1235.608788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1235.879830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.907122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.935703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.957843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.986400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.011198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.039935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.064787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.090207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.131510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.182490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.207606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.233836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.255971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.286779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.787185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1236.821861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1236.873196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1236.908490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1236.941957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 1 [ 1236.998963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 1 [ 1237.090800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 1 [ 1237.155367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 1 [ 1237.224765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1237.253439] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1237.259888] LNet: Skipped 5 previous similar messages [ 1237.479052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.507100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.533754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.583572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.619093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.646422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.676383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.711856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.726806] LNet: 61337:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.731892] LNet: 61337:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1237.746665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.789708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.829866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.860978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.907894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.952772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.991374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.508991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1238.543125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1238.586092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1238.611661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1238.645034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_error -r 1 [ 1238.690117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_error -r 1 [ 1238.735515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_error -r 1 [ 1238.807910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_error -r 1 [ 1238.858281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1239.156950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.209843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.269507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.309344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.373431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.434587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.494422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.539612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.570059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.596740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.644557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.689266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.726509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.772990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.818083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.975197] LNet: 61534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.981603] LNetError: 61534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.986992] LNet: Removed LNI 192.168.204.42@tcp [ 1248.052357] LNet: Removed LNI 192.168.204.42@tcp1 [ 1248.569627] Key type .llcrypt unregistered [ 1248.573981] Key type ._llcrypt unregistered [ 1266.919591] Key type ._llcrypt registered [ 1266.921567] Key type .llcrypt registered [ 1268.188325] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 11:29:02 (1788449342) [ 1269.301413] Key type .llcrypt unregistered [ 1269.303154] Key type ._llcrypt unregistered [ 1275.972271] Key type ._llcrypt registered [ 1275.974704] Key type .llcrypt registered [ 1276.145952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.156680] alg: No test for adler32 (adler32-zlib) [ 1277.074711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1277.109033] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1284.498119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1285.356140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1285.386901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1285.425514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1285.442705] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1285.711937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.781488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1285.833538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.892800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.953596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.026841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.090608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.147948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.183497] LNet: 62778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.213461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.250343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.295921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.348909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.422934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.519484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.562724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.687410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1287.728887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1287.803970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1287.833178] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1288.118743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.156499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.194545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.246880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.311257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.386697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.425105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.460877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.473929] LNet: 62888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.487899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.532469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.603514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.662920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.754490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.788506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.830736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1289.578807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1289.645904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1289.730173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1289.766240] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1290.100580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.140730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.202817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.279506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.335174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.405473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.466971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.518949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.545024] LNet: 62997:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.560801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.601304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.641316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.671821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.705884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.782560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.816765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1291.133231] LNet: 63038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.156434] LNetError: 63038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.202694] LNet: Removed LNI 192.168.204.42@tcp [ 1291.782645] Key type .llcrypt unregistered [ 1291.785970] Key type ._llcrypt unregistered [ 1310.401270] Key type ._llcrypt registered [ 1310.405670] Key type .llcrypt registered [ 1311.776687] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 11:29:46 (1788449386) [ 1312.966565] Key type .llcrypt unregistered [ 1312.970555] Key type ._llcrypt unregistered [ 1319.364072] Key type ._llcrypt registered [ 1319.365644] Key type .llcrypt registered [ 1319.640930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.678964] alg: No test for adler32 (adler32-zlib) [ 1320.624442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.690145] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1327.831874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1336.507981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.556571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1336.596566] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1338.078735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1338.132649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1338.176242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1338.233890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1338.278934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 1 [ 1338.318646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 1 [ 1338.360863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 1 [ 1338.406757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 1 [ 1338.450751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1338.481949] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1338.485882] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1338.696687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.727557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.758166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.791904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.827290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.867491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.896942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.926088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.938596] LNet: 64501:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.953924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.985201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.020579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.065420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.099833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.140314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.194700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1339.254684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.802679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1339.851908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1339.914741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1339.982422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1340.040322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_error -r 1 [ 1340.082679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_error -r 1 [ 1340.124553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_error -r 1 [ 1340.155313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_error -r 1 [ 1340.183908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1340.200586] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1340.205547] LNet: Skipped 2 previous similar messages [ 1340.411708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.443299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1340.471721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.508619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.537512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.569926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1340.616508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1340.649934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1340.664149] LNet: 64624:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1340.678722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1340.706184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1340.732531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1340.766056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1340.802434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1340.835686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1340.884367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1340.920515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.442750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1341.474204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1341.505540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1341.536376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1341.576652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_timeout -r 1 [ 1341.622038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_timeout -r 1 [ 1341.663234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_timeout -r 1 [ 1341.695559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_timeout -r 1 [ 1341.742763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.142@tcp [ 1341.761900] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1341.999138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.047389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.072660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.116366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.170676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.206368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.249361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.304176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.331744] LNet: 64747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.346784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.370107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.433985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.463659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.489943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.533721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1342.570696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.597412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.342424] LNet: 64825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.348074] LNetError: 64825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.360996] LNet: Removed LNI 192.168.204.42@tcp [ 1347.379852] LNet: Removed LNI 192.168.204.42@tcp1 [ 1347.792656] Key type .llcrypt unregistered [ 1347.795969] Key type ._llcrypt unregistered [ 1362.781123] Key type ._llcrypt registered [ 1362.785364] Key type .llcrypt registered [ 1364.244739] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 11:30:38 (1788449438) [ 1365.860108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1367.790126] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 11:30:42 (1788449442) [ 1368.797608] Key type .llcrypt unregistered [ 1368.799114] Key type ._llcrypt unregistered [ 1375.528437] Key type ._llcrypt registered [ 1375.532480] Key type .llcrypt registered [ 1375.733347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.749896] alg: No test for adler32 (adler32-zlib) [ 1376.696386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1376.726228] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1383.502928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1384.229810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1384.345433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1384.404426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1384.424979] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1384.855636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1384.933936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1384.938778] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1385.034817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1385.113300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1385.171464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1385.195995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1385.242106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1385.271687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1385.296265] LNet: 66363:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1385.332833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1385.373628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.416704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.446501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.478796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.511707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1385.758130] LNet: 66402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.774784] LNetError: 66402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.789507] LNet: Removed LNI 192.168.204.42@tcp [ 1386.171877] Key type .llcrypt unregistered [ 1386.174759] Key type ._llcrypt unregistered [ 1404.710545] Key type ._llcrypt registered [ 1404.712969] Key type .llcrypt registered [ 1404.912543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1404.938846] alg: No test for adler32 (adler32-zlib) [ 1405.843833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1405.892857] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1414.054336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1423.745511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.802862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.840726] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1425.178668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1425.248187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1425.314821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1425.391658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1425.455861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e network_timeout -r 1 [ 1425.489281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e network_timeout -r 1 [ 1425.522854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e network_timeout -r 1 [ 1425.555092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e network_timeout -r 1 [ 1425.587800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1425.603315] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1425.803451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1425.857834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1425.888294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1425.915745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1425.937246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1425.967315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1426.015529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1426.061470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1426.077688] LNet: 67276:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1426.122109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1426.198840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1426.266803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.335115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1426.384107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1426.416334] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1426.416438] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1426.424970] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1426.440982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1426.447053] LNet: Skipped 1 previous similar message [ 1428.448395] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1430.498261] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1432.460966] LNet: 67350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.474490] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.499083] LNet: Removed LNI 192.168.204.42@tcp [ 1432.518686] LNet: Removed LNI 192.168.204.42@tcp1 [ 1432.918861] Key type .llcrypt unregistered [ 1432.920516] Key type ._llcrypt unregistered [ 1450.882415] Key type ._llcrypt registered [ 1450.884059] Key type .llcrypt registered [ 1452.243755] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 11:32:06 (1788449526) [ 1452.698358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.717668] alg: No test for adler32 (adler32-zlib) [ 1453.683397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.753215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.782864] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1454.838777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.877420] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1455.869518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1455.986045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1456.035820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1456.066769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1456.103702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1456.145342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1456.183662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1456.208666] LNetError: 68448:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1456.799405] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1456.799615] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1458.848128] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1459.699801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1462.943923] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1463.210111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1463.251679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1463.339780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.369605] LNet: 68555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.378989] LNetError: 68555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.386135] LNet: Removed LNI 192.168.204.42@tcp [ 1463.423201] LNet: Removed LNI 192.168.204.42@tcp1 [ 1463.443523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.526123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.552587] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1464.423159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.462454] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1465.314487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1465.390516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.473609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1465.517624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1465.572555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1465.638729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1465.683484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1465.712786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1465.730600] LNetError: 68828:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1466.656626] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1469.313661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.803088] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1472.814205] LNet: Skipped 1 previous similar message [ 1477.257297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1477.307771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1477.361846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.436123] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 11:32:39 (1788449559) [ 1485.531608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.556429] LNet: 69543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.562154] LNetError: 69543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.566985] LNetError: 69543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1485.572318] LNet: Removed LNI 192.168.204.42@tcp [ 1485.608598] LNet: Removed LNI 192.168.204.42@tcp1 [ 1485.637828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.799671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.864346] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1486.670183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.691198] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1487.845645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1487.910588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1487.941605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1487.976957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1488.018908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1488.069693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1488.107248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1488.268895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1488.863684] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.42@tcp [ 1489.888967] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1489.904315] LNet: Skipped 1 previous similar message [ 1504.236350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.389581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1504.437218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1504.488790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 500 [ 1504.613726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.406994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.526745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.559661] LNet: 70040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.592416] LNetError: 70040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.606872] LNetError: 70040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1511.613718] LNet: Removed LNI 192.168.204.42@tcp [ 1511.636214] LNet: Removed LNI 192.168.204.42@tcp1 [ 1511.672339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.767758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.814812] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1512.804138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.850486] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1514.085759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1514.164358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1514.277770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1514.319418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1514.355138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1514.399756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1514.433902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1514.466466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.42@tcp --health 0 [ 1515.807450] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1515.812894] LNet: Skipped 3 previous similar messages [ 1516.829317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.802798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.832718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1525.864040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1533.916956] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 11:33:28 (1788449608) [ 1534.605918] LNet: 71090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.618266] LNetError: 71090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.623924] LNetError: 71090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1534.638443] LNet: Removed LNI 192.168.204.42@tcp [ 1534.666731] LNet: Removed LNI 192.168.204.42@tcp1 [ 1535.504941] Key type .llcrypt unregistered [ 1535.507523] Key type ._llcrypt unregistered [ 1541.184318] Key type ._llcrypt registered [ 1541.186967] Key type .llcrypt registered [ 1541.460049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1541.487866] alg: No test for adler32 (adler32-zlib) [ 1542.434653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1542.469346] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1552.148697] Lustre: DEBUG MARKER: Initial discovery [ 1552.310544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1554.667840] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1554.807819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1554.853212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1554.887475] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1554.927379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1554.990458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -l 3 [ 1555.021444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.142@tcp -r 1 -m GET -e local_error [ 1555.056477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.142@tcp -r 1 -m PUT -l 6 [ 1555.807292] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1556.651460] Lustre: DEBUG MARKER: Force oleg442-client.virtnet to discover 192.168.204.142@tcp (in background) [ 1556.868724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.142@tcp [ 1558.389844] Lustre: DEBUG MARKER: Force oleg442-server to discover 192.168.204.42@tcp [ 1559.968054] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1560.894293] Lustre: DEBUG MARKER: Wait for 71847 [ 1562.506898] Lustre: DEBUG MARKER: Finished wait on 71847 [ 1562.558285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1562.591876] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1562.608103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1562.885366] LNet: 72299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.895156] LNetError: 72299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.936979] LNet: Removed LNI 192.168.204.42@tcp [ 1563.434685] Key type .llcrypt unregistered [ 1563.437883] Key type ._llcrypt unregistered [ 1582.705903] Key type ._llcrypt registered [ 1582.712168] Key type .llcrypt registered [ 1584.554545] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 11:34:18 (1788449658) [ 1585.412306] Key type .llcrypt unregistered [ 1585.414964] Key type ._llcrypt unregistered [ 1593.093694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1593.259559] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1593.273314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1593.280662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1596.499595] Key type ._llcrypt registered [ 1596.504137] Key type .llcrypt registered [ 1596.964646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1597.020617] alg: No test for adler32 (adler32-zlib) [ 1597.990344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.153705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.241438] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1599.782641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1599.826860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1613.954878] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:34:47 (1788449687) [ 1614.133608] LNet: 74363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.157122] LNetError: 74363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1614.185693] LNet: Removed LNI 10.1.2.3@tcp [ 1614.205576] LNet: Removed LNI 192.168.204.42@tcp [ 1614.802098] Key type .llcrypt unregistered [ 1614.804639] Key type ._llcrypt unregistered [ 1621.575639] Key type ._llcrypt registered [ 1621.578354] Key type .llcrypt registered [ 1621.897837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1621.935660] alg: No test for adler32 (adler32-zlib) [ 1622.831818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.878650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.904685] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1623.827587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1623.867276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1625.365681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.42@tcp [ 1626.399917] LNetError: Unexpected error -22 connecting to 192.168.204.42@tcp at host 192.168.204.42:988 [ 1632.634069] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:35:07 (1788449707) [ 1632.700596] LNet: 75490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1632.711618] LNetError: 75490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1632.724249] LNet: Removed LNI 10.1.2.3@tcp [ 1632.742266] LNet: Removed LNI 192.168.204.42@tcp [ 1633.387368] Key type .llcrypt unregistered [ 1633.389681] Key type ._llcrypt unregistered [ 1640.323652] Key type ._llcrypt registered [ 1640.325594] Key type .llcrypt registered [ 1640.530222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1640.550624] alg: No test for adler32 (adler32-zlib) [ 1641.529743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1641.608633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1641.640046] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1642.356904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1642.385874] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 1643.517575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.42@tcp1 --nid 192.168.204.42@tcp2 [ 1647.624246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp2 [ 1647.651120] LNetError: 76332:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.42@tcp1 to 192.168.204.42@tcp2 - Route table may be misconfigured [ 1647.681734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp2 192.168.204.42@tcp1 [ 1653.719777] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:35:28 (1788449728) [ 1654.084645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.120788] LNet: 76907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.128742] LNetError: 76907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.135893] LNet: Removed LNI 192.168.204.42@tcp1 [ 1655.216498] LNet: Removed LNI 192.168.204.42@tcp2 [ 1655.241436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.303486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.358481] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1656.051127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.068861] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1656.925955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 1656.986736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.42@tcp [ 1657.023094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp1 192.168.204.42@tcp1 [ 1657.062391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp -e network_timeout [ 1657.094381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -e network_timeout [ 1657.135684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -e network_timeout [ 1657.166918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -e network_timeout [ 1657.196646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1657.217658] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1657.236751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1657.311326] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1657.352829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1657.464787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1663.381102] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:35:38 (1788449738) [ 1663.732839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.762819] LNet: 77768:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.770552] LNetError: 77768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.774939] LNetError: 77768:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1663.779779] LNet: Removed LNI 192.168.204.42@tcp [ 1664.806980] LNet: Removed LNI 192.168.204.42@tcp1 [ 1664.856884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.947304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1665.228084] LNet: 77806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.543776] Key type .llcrypt unregistered [ 1665.547171] Key type ._llcrypt unregistered [ 1675.499089] Key type ._llcrypt registered [ 1675.503070] Key type .llcrypt registered [ 1676.621464] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:35:51 (1788449751) [ 1677.142959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.165587] alg: No test for adler32 (adler32-zlib) [ 1678.026228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.196229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1680.758910] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:35:55 (1788449755) [ 1680.844792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.886600] LNet: 78896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.936486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.998920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1681.019782] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1681.731430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1681.755227] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1682.506349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1682.539904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp1 [ 1682.575968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp1 [ 1688.926347] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:36:03 (1788449763) [ 1690.789553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg442-server [ 1692.483591] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:36:07 (1788449767) [ 1694.856511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg442-server [ 1696.551913] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:36:11 (1788449771) [ 1699.018661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg442-server [ 1700.726757] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:36:15 (1788449775) [ 1702.326584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg442-server [ 1703.965736] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:36:18 (1788449778) [ 1706.363862] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg442-server [ 1707.998697] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:36:22 (1788449782) [ 1709.681213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg442-server [ 1711.385812] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:36:26 (1788449786) [ 1713.622485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg442-server [ 1715.799501] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:36:30 (1788449790) [ 1717.873615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg442-server [ 1719.983881] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:36:34 (1788449794) [ 1722.857276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg442-server [ 1725.460986] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:36:39 (1788449799) [ 1725.781330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.810123] LNet: 82781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.823451] LNetError: 82781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.843704] LNet: Removed LNI 192.168.204.42@tcp [ 1726.888706] LNet: Removed LNI 192.168.204.42@tcp1 [ 1726.983211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.062287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.083911] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1727.915103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1728.059993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1729.497445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.529762] LNet: 82937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.537755] LNetError: 82937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.559746] LNetError: 82937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1729.602883] LNet: Removed LNI 192.168.204.42@tcp [ 1729.663181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.773102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1729.851065] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1730.938878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1731.257079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.363841] LNetError: 83083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.406024] LNet: Removed LNI 192.168.204.42@tcp [ 1732.429591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.495768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.564917] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1733.679321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1733.748542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1733.976593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.006411] LNet: 83230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.015035] LNet: 83230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1734.031163] LNetError: 83230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.078575] LNet: Removed LNI 192.168.204.42@tcp [ 1735.128932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.260221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1735.344121] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1736.511297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1736.814917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.903223] LNet: Removed LNI 192.168.204.42@tcp [ 1737.978661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1738.113648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1738.164842] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1738.993879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1739.096524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1740.567571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.628776] LNet: 83532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.638380] LNet: 83532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1740.648430] LNetError: 83532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.662166] LNetError: 83532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1740.677899] LNet: Removed LNI 192.168.204.42@tcp [ 1740.697844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.764736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1740.801855] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1741.740330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1742.046421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.142487] LNet: Removed LNI 192.168.204.42@tcp [ 1743.201558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.322703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.358962] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1744.005713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1744.129295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1745.677077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.737025] LNet: Removed LNI 192.168.204.42@tcp [ 1746.796492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.858099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1746.904211] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1747.711358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1747.906190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.979687] LNet: Removed LNI 192.168.204.42@tcp [ 1749.042560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.079521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.097191] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1750.017933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1750.116777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1750.392462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.410437] LNet: 84126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.414909] LNet: 84126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1750.420129] LNetError: 84126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.424818] LNetError: 84126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1751.494382] LNet: Removed LNI 192.168.204.42@tcp [ 1751.548097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.673712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1751.715315] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1752.818085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1754.431624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.508770] LNet: Removed LNI 192.168.204.42@tcp [ 1755.559749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.654576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.694305] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1756.553588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1756.649873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1756.953781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.002807] LNet: Removed LNI 192.168.204.42@tcp [ 1758.048388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.136992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1758.190416] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1758.984969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 1759.277464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.358583] LNet: Removed LNI 192.168.204.42@tcp [ 1760.395301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.464483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.491630] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1761.359437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1761.413446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.445618] LNet: Removed LNI 192.168.204.42@tcp [ 1761.463633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.527565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.564272] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1762.458465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1762.550893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.589348] LNet: Removed LNI 192.168.204.42@tcp [ 1762.625366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.689163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1762.790316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.847061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.926994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1763.012572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.050715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.146630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1763.287777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.342244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.428286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1763.467569] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1764.241360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.302167] LNet: Removed LNI 192.168.204.42@tcp [ 1764.322870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.418862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1771.110411] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:37:26 (1788449846) [ 1771.182469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.222324] LNet: 85588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1771.228664] LNet: 85588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1771.253494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.308847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1771.330856] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1772.382535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.404656] LNetError: 85732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.409406] LNetError: 85732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1772.419465] LNet: Removed LNI 192.168.204.42@tcp231 [ 1772.438587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.493520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1772.511462] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1772.516481] LNet: 85740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.805577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1772.837284] LNet: Removed LNI 192.168.204.42@tcp231 [ 1772.856284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1772.899264] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1773.826614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.860434] LNet: Removed LNI 192.168.204.42@tcp231 [ 1773.898681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.959546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1773.982211] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1773.988648] LNet: 85893:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.301414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1774.329895] LNet: Removed LNI 192.168.204.42@tcp231 [ 1774.347852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1774.371732] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/0] [ 1775.273657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.304819] LNet: Removed LNI 192.168.204.42@tcp231 [ 1775.326050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.403671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1775.440085] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1775.451779] LNet: 86044:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1775.946529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1775.972195] LNet: Removed LNI 192.168.204.42@tcp231 [ 1775.997425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1776.030706] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/60] [ 1776.954135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.999198] LNet: Removed LNI 192.168.204.42@tcp231 [ 1777.034104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.112178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1777.131940] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1777.604727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1777.629768] LNet: Removed LNI 192.168.204.42@tcp231 [ 1777.648745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1777.695719] LNet: Added LNI 192.168.204.42@tcp231 [8/256/0/180] [ 1783.808869] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:37:38 (1788449858) [ 1784.004508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.075200] LNet: Removed LNI 192.168.204.42@tcp231 [ 1784.138887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.212451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1784.237671] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1785.098842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1785.146496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1785.191625] LNet: Removed LNI 192.168.204.42@tcp232 [ 1785.224439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1785.265388] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1785.846318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1785.885423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1785.914539] LNet: Removed LNI 192.168.204.42@tcp232 [ 1785.929807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1785.954632] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1786.547911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1786.587890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1786.615935] LNet: Removed LNI 192.168.204.42@tcp232 [ 1786.633865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1786.670949] LNet: Added LNI 192.168.204.42@tcp232 [8/256/0/180] [ 1787.388759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1787.418417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1787.441711] LNet: Removed LNI 192.168.204.42@tcp232 [ 1787.459741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1793.046386] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:37:47 (1788449867) [ 1794.185824] Key type .llcrypt unregistered [ 1794.188103] Key type ._llcrypt unregistered [ 1799.480242] Key type ._llcrypt registered [ 1799.483397] Key type .llcrypt registered [ 1799.659217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1799.670829] alg: No test for adler32 (adler32-zlib) [ 1800.584779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1800.610751] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1808.229536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1817.313716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.378659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1817.423403] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1818.846940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 1818.900934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 1818.936413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 1818.963466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 1819.009531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_interrupt -r 2 [ 1819.055856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_interrupt -r 2 [ 1819.102841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_interrupt -r 2 [ 1819.165629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_interrupt -r 2 [ 1819.219658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1819.247669] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1819.255428] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1819.484445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.513903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.547794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.573603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.600733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.639645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.675143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.713696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.740917] LNet: 88693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.776191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.827595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.870799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.908548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.945894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.975399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.536641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 1820.570886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 1820.609630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 1820.649687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 1820.688131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_dropped -r 2 [ 1820.722835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_dropped -r 2 [ 1820.757387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_dropped -r 2 [ 1820.800626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_dropped -r 2 [ 1820.838277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1820.857899] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1821.059894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.082940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1821.108583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.139870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.177775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.214159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.255314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.307393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.331333] LNet: 88813:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.351952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1821.382418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1821.417597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1821.451108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1821.491479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1821.514708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1821.970786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 1822.006603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 1822.037296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 1822.084333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 1822.139197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_aborted -r 2 [ 1822.181353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_aborted -r 2 [ 1822.231915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_aborted -r 2 [ 1822.280987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_aborted -r 2 [ 1822.312968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1822.344107] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1822.574503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1822.607330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1822.654555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1822.686847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1822.715656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1822.748274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1822.781868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1822.809622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1822.825153] LNet: 88934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.842113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1822.870264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1822.902400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.934060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1822.970654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1823.007646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1823.578135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 1823.614700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 1823.651388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 1823.703326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 1823.749414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_no_route -r 2 [ 1823.784406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_no_route -r 2 [ 1823.826697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_no_route -r 2 [ 1823.884936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_no_route -r 2 [ 1823.933163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1824.229338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1824.267348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1824.300476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1824.370813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1824.404389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1824.466406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1824.513448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1824.561469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1824.606027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1824.664850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1824.720637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1824.755981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1824.806581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1824.853142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1825.282617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 1825.324916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 1825.371255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 1825.403461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 1825.455497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e local_timeout -r 2 [ 1825.514961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e local_timeout -r 2 [ 1825.566614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e local_timeout -r 2 [ 1825.610600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e local_timeout -r 2 [ 1825.648721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1825.669630] LNet: There was an unexpected network error while writing to 192.168.204.142: rc = -22 [ 1825.675395] LNet: Skipped 1 previous similar message [ 1825.921699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1825.954303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1825.981951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1826.008717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1826.035580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1826.060296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1826.101849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1826.137719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1826.158953] LNet: 89174:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.167337] LNet: 89174:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1826.183572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1826.232371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1826.290637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1826.338390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1826.380284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1826.412444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1826.895841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 1826.921742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 1826.976816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 1827.041620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 1827.081856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp -m GET -e remote_dropped -r 2 [ 1827.108362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.142@tcp1 -m GET -e remote_dropped -r 2 [ 1827.171751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp -m GET -e remote_dropped -r 2 [ 1827.225670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.42@tcp1 -d 192.168.204.42@tcp1 -m GET -e remote_dropped -r 2 [ 1827.276596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.42@tcp 192.168.204.142@tcp [ 1827.304625] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 1827.563296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1827.594695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1827.620649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1827.672326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1827.714522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1827.767907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1827.843414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1827.903694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1827.975982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1828.027556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1828.101379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1828.185331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1828.235299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1828.283138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1833.884349] LNet: 89369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.895516] LNetError: 89369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.904103] LNet: Removed LNI 192.168.204.42@tcp [ 1833.919667] LNet: Removed LNI 192.168.204.42@tcp1 [ 1834.260553] Key type .llcrypt unregistered [ 1834.267387] Key type ._llcrypt unregistered [ 1849.049260] Key type ._llcrypt registered [ 1849.052500] Key type .llcrypt registered [ 1850.223212] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:38:44 (1788449924) [ 1850.700000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.715254] alg: No test for adler32 (adler32-zlib) [ 1851.589793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.653083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.690155] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1852.398014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1852.425322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.42@tcp [ 1852.446578] LNetError: 90345:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1852.534950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.42@tcp [ 1852.646231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.312336] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:38:54 (1788449934) [ 1859.574187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.599625] LNet: 90936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.607725] LNetError: 90936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.633458] LNet: Removed LNI 192.168.204.42@tcp [ 1859.648056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.730717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1859.765225] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1860.475954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1860.502754] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 1861.669738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.42@tcp [ 1861.690749] LNetError: 91202:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.42@tcp to recovery [ 1861.727573] LNet: 1 local NIs in recovery (showing 1): 192.168.204.42@tcp [ 1861.729934] LNet: There was an unexpected network error while writing to 192.168.204.42: rc = -22 [ 1861.866661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.42@tcp [ 1870.119179] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:39:05 (1788449945) [ 1872.117301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg442-server [ 1873.803443] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:39:08 (1788449948) [ 1875.510250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg442-server [ 1877.079707] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:39:11 (1788449951) [ 1878.891331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg442-server [ 1880.549157] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:39:15 (1788449955) [ 1880.764797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1880.808973] LNet: 92823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.826718] LNetError: 92823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1880.838569] LNet: Removed LNI 192.168.204.42@tcp [ 1880.866797] LNet: Removed LNI 192.168.204.42@tcp1 [ 1880.941861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.993654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.012430] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1881.680656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1881.731557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1881.783712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1881.932146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1881.993308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1882.075712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1888.186184] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:39:23 (1788449963) [ 1888.543926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1888.600232] LNet: 93554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.615280] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.625770] LNetError: 93554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1888.640111] LNet: Removed LNI 192.168.204.42@tcp [ 1888.662711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.721349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1888.748315] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1895.300572] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:39:30 (1788449970) [ 1895.421542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.446982] LNet: 94252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.456135] LNetError: 94252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.480524] LNet: Removed LNI 192.168.204.42@tcp [ 1895.517253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.609389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.647036] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1902.192537] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:39:37 (1788449977) [ 1903.766549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1905.208301] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:39:40 (1788449980) [ 1905.611345] LNet: 95292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.618292] LNetError: 95292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.633810] LNet: Removed LNI 192.168.204.42@tcp [ 1906.210225] Key type .llcrypt unregistered [ 1906.213104] Key type ._llcrypt unregistered [ 1911.911951] Key type ._llcrypt registered [ 1911.914391] Key type .llcrypt registered [ 1912.125678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1912.144612] alg: No test for adler32 (adler32-zlib) [ 1913.036898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1913.071508] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1919.758489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 1923.863597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.142@tcp [ 1925.313424] LNet: 95707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.327578] LNetError: 95707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.363989] LNet: Removed LNI 192.168.204.42@tcp [ 1925.765571] Key type .llcrypt unregistered [ 1925.774862] Key type ._llcrypt unregistered [ 1935.938987] Key type ._llcrypt registered [ 1935.941988] Key type .llcrypt registered [ 1937.724505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1939.855933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1941.759046] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:40:16 (1788450016) [ 1943.747940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1945.788180] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:40:20 (1788450020) [ 1947.725664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg442-server [ 1949.758466] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:40:24 (1788450024) [ 1951.736230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg442-server [ 1953.732001] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:40:28 (1788450028) [ 1955.200060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.323554] alg: No test for adler32 (adler32-zlib) [ 1956.374692] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 1956.747967] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 1958.431841] Key type lgssc registered [ 1959.636636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1979.942306] Key type lgssc unregistered [ 1980.307552] LNet: 98876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.317217] LNetError: 98876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.336275] LNet: Removed LNI 192.168.204.42@tcp [ 1981.232723] Key type .llcrypt unregistered [ 1981.237081] Key type ._llcrypt unregistered [ 1997.160743] Key type ._llcrypt registered [ 1997.162857] Key type .llcrypt registered [ 1998.501888] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:41:13 (1788450073) [ 1999.039000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.059893] alg: No test for adler32 (adler32-zlib) [ 1999.960335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.041843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2000.090413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2000.139527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2000.180626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2006.475165] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:41:21 (1788450081) [ 2006.623613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.653732] LNet: 100289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.674454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.728302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2006.753942] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2007.508338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.533978] LNet: 100433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.545399] LNetError: 100433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.576210] LNet: Removed LNI 192.168.204.42@tcp [ 2007.601835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.655035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2007.693670] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2008.150297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.204910] LNetError: 100461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.219415] LNet: Removed LNI 192.168.204.42@tcp [ 2008.240302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.289869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2008.314243] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2014.623569] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:41:29 (1788450089) [ 2014.823755] LNet: 101051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2014.828836] LNet: 101051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2014.835401] LNetError: 101051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.850574] LNet: Removed LNI 192.168.204.42@tcp [ 2015.480931] Key type .llcrypt unregistered [ 2015.486060] Key type ._llcrypt unregistered [ 2021.412524] Key type ._llcrypt registered [ 2021.414937] Key type .llcrypt registered [ 2021.672434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2021.691569] alg: No test for adler32 (adler32-zlib) [ 2022.614900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.676225] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2022.678875] LNet: *** cfs_fail_loc=e003, val=100*** [ 2023.510564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2023.526682] LNet: *** cfs_fail_loc=e003, val=100*** [ 2023.530967] LNet: Skipped 2 previous similar messages [ 2023.535343] Ping reply received with mapping: [ 2023.536987] found metadata reply NI[192.168.204.42@tcp] [ 2023.540295] 2a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2023.543772] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2023.549312] 5a 5a 5a 5a 5a 5a 5a 5a [ 2023.667753] LNet: 101444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.677659] LNetError: 101444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2024.741384] LNet: Removed LNI 192.168.204.42@tcp [ 2025.506702] Key type .llcrypt unregistered [ 2025.510771] Key type ._llcrypt unregistered [ 2039.003526] Key type ._llcrypt registered [ 2039.006369] Key type .llcrypt registered [ 2040.724643] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:41:54 (1788450114) [ 2041.321336] Key type .llcrypt unregistered [ 2041.323382] Key type ._llcrypt unregistered [ 2048.625521] Key type ._llcrypt registered [ 2048.631799] Key type .llcrypt registered [ 2048.894941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2048.910638] alg: No test for adler32 (adler32-zlib) [ 2050.127193] LNetError: 102483:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2050.610887] Key type .llcrypt unregistered [ 2050.618827] Key type ._llcrypt unregistered [ 2056.784298] Key type ._llcrypt registered [ 2056.787860] Key type .llcrypt registered [ 2058.291389] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:42:12 (1788450132) [ 2058.917523] Key type .llcrypt unregistered [ 2058.924161] Key type ._llcrypt unregistered [ 2066.376225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2066.382171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2069.132564] Key type ._llcrypt registered [ 2069.135116] Key type .llcrypt registered [ 2069.388882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2069.410245] alg: No test for adler32 (adler32-zlib) [ 2070.346724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.429464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2080.949341] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:42:35 (1788450155) [ 2081.234488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.286780] LNet: 104257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.297401] LNetError: 104257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2081.340752] LNet: Removed LNI 10.1.2.3@tcp [ 2081.365360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.086309] Key type .llcrypt unregistered [ 2082.089585] Key type ._llcrypt unregistered [ 2089.454372] Key type ._llcrypt registered [ 2089.460952] Key type .llcrypt registered [ 2089.716882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2089.749099] alg: No test for adler32 (adler32-zlib) [ 2090.933975] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2091.042432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2091.251127] LNet: 104542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.259681] LNetError: 104542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.294858] LNet: Removed LNI 192.168.204.42@tcp [ 2092.149576] Key type .llcrypt unregistered [ 2092.154964] Key type ._llcrypt unregistered [ 2105.813569] Key type ._llcrypt registered [ 2105.816303] Key type .llcrypt registered [ 2107.191709] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:43:01 (1788450181) [ 2107.994296] Key type .llcrypt unregistered [ 2107.996200] Key type ._llcrypt unregistered [ 2114.898236] Key type ._llcrypt registered [ 2114.904314] Key type .llcrypt registered [ 2115.112605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2115.154129] alg: No test for adler32 (adler32-zlib) [ 2122.836843] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:43:17 (1788450197) [ 2123.199132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.260502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.362616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.411838] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2124.248439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.274346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2124.302092] LNet: Removed LNI 192.168.204.42@tcp [ 2124.532397] LNet: 106282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.937793] Key type .llcrypt unregistered [ 2124.947243] Key type ._llcrypt unregistered [ 2137.860854] Key type ._llcrypt registered [ 2137.864713] Key type .llcrypt registered [ 2139.164278] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:43:33 (1788450213) [ 2139.659222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2139.676202] alg: No test for adler32 (adler32-zlib) [ 2140.617179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.693125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.719189] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2141.598986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2141.629377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.42@tcp [ 2141.641946] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2141.645302] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2141.648191] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2141, 0, 0 [ 2141.652284] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2141.658194] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2141.662211] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2141.665606] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2141, 0, 0 [ 2141.672989] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2141.675844] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2141.679538] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2141.683179] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2141, 0, 2191 [ 2141.688227] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2141.690949] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2141.694984] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2141.697641] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2141, 0, 2191 [ 2141.702543] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2149.179870] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:43:43 (1788450223) [ 2149.657457] LNet: 107819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.668182] LNetError: 107819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.694340] LNet: Removed LNI 192.168.204.42@tcp [ 2151.483445] Key type .llcrypt unregistered [ 2151.492989] Key type ._llcrypt unregistered [ 2159.659918] Key type ._llcrypt registered [ 2159.662482] Key type .llcrypt registered [ 2160.006909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2160.026213] alg: No test for adler32 (adler32-zlib) [ 2161.016224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2161.046272] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2169.118574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2179.059462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.115451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2179.140722] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2180.234398] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2180.360923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2180.364707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2180.371293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2182.737669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2182.772163] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2194.698824] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:44:29 (1788450269) [ 2195.066376] LNet: 109326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.076887] LNetError: 109326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.087771] LNet: Removed LNI 192.168.204.42@tcp [ 2196.128331] LNet: Removed LNI 192.168.204.42@tcp1 [ 2196.139226] LNetError: 109326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2196.148867] LNetError: 109326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2196.175490] LNet: Removed LNI 10.1.2.3@tcp99 [ 2196.905076] Key type .llcrypt unregistered [ 2196.907455] Key type ._llcrypt unregistered [ 2205.045462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2205.063457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2207.833444] Key type ._llcrypt registered [ 2207.835696] Key type .llcrypt registered [ 2208.063792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2208.087923] alg: No test for adler32 (adler32-zlib) [ 2209.186754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.288419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.325902] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2209.954216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2209.989846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2213.441875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2213.487470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2213.526845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2213.547847] LNetError: 110067:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2213.579210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2213.612785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2214.367728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2214.478571] LNet: 110088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.483732] LNetError: 110088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2214.490160] LNet: Removed LNI 10.1.2.3@tcp [ 2215.526499] LNet: Removed LNI 192.168.204.42@tcp [ 2216.533682] Key type .llcrypt unregistered [ 2216.541991] Key type ._llcrypt unregistered [ 2231.279766] Key type ._llcrypt registered [ 2231.282687] Key type .llcrypt registered [ 2232.578224] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:45:07 (1788450307) [ 2233.095087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2233.123776] alg: No test for adler32 (adler32-zlib) [ 2234.084601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.193756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.260799] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2235.147962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2235.235134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg442-client@tcp [ 2242.554433] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:45:17 (1788450317) [ 2242.719169] LNet: 111610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.723439] LNetError: 111610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.755063] LNet: Removed LNI 192.168.204.42@tcp [ 2243.529386] Key type .llcrypt unregistered [ 2243.535906] Key type ._llcrypt unregistered [ 2250.572180] Key type ._llcrypt registered [ 2250.580522] Key type .llcrypt registered [ 2250.757778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2250.768696] alg: No test for adler32 (adler32-zlib) [ 2251.728234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2251.773519] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2251.808455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2258.337390] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:45:32 (1788450332) [ 2259.942802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2261.552933] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:45:36 (1788450336) [ 2263.031846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2264.823877] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:45:39 (1788450339) [ 2264.980708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.005459] LNet: 113037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.012231] LNetError: 113037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.025240] LNet: Removed LNI 192.168.204.42@tcp [ 2265.041573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.090582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.128800] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2265.831921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2265.879652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2272.324589] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:45:47 (1788450347) [ 2272.550604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.580812] LNet: 113746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.603617] LNetError: 113746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.622466] LNet: Removed LNI 192.168.204.42@tcp [ 2272.674861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.796070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2272.882484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2273.717548] Key type .llcrypt unregistered [ 2273.720648] Key type ._llcrypt unregistered [ 2285.876255] Key type ._llcrypt registered [ 2285.880743] Key type .llcrypt registered [ 2287.500891] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:46:01 (1788450361) [ 2287.928578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2287.939379] alg: No test for adler32 (adler32-zlib) [ 2289.004469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.107734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.147709] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2290.085237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2290.252117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2299.484147] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:46:13 (1788450373) [ 2299.581371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2299.618307] LNet: 115291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.628157] LNetError: 115291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.673154] LNet: Removed LNI 192.168.204.42@tcp [ 2299.736649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.796765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2299.840946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2307.378774] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:46:21 (1788450381) [ 2307.615716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.656319] LNet: 115879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.684684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2307.792700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2308.276049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2308.316868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2308.391953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2308.698186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2308.725633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2308.800952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2308.835177] LNet: 115913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.857777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2308.928368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2309.271786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2309.307417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2309.380505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2309.721397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2309.772921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2317.583850] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:46:31 (1788450391) [ 2317.918561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.951879] LNet: 116513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.983980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.034173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2318.060447] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2318.880633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2318.895909] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2319.876576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2319.932427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2320.166867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.186809] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.194804] LNet: Removed LNI 192.168.204.42@tcp [ 2320.218580] LNet: Removed LNI 192.168.204.42@tcp2 [ 2320.248154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.309178] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2320.314403] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2320.318948] LNet: 116798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2320.855865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2321.105564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.207248] LNet: Removed LNI 192.168.204.42@tcp [ 2322.211798] LNetError: 116836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.216216] LNetError: 116836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2322.232929] LNet: Removed LNI 192.168.204.42@tcp2 [ 2322.256675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.367126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.381982] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2323.048319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2323.073788] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2324.231609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2324.278550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2324.351835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2324.572194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.603833] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.623280] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2324.639904] LNetError: 117120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2324.657513] LNet: Removed LNI 192.168.204.42@tcp [ 2325.674662] LNet: Removed LNI 192.168.204.42@tcp2 [ 2325.739848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.801725] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2325.810022] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2325.827320] LNet: 117126:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2326.630034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2326.865439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.899284] LNet: Removed LNI 192.168.204.42@tcp [ 2327.977533] LNet: Removed LNI 192.168.204.42@tcp2 [ 2328.056414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.187689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.227987] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2329.117464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2329.136931] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2329.833672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2329.883110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2330.086586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.109417] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.115492] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2330.122718] LNet: Removed LNI 192.168.204.42@tcp [ 2330.135157] LNet: Removed LNI 192.168.204.42@tcp2 [ 2330.155327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.209403] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2330.215943] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2330.221640] LNet: 117449:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.594378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2330.874316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.935142] LNet: Removed LNI 192.168.204.42@tcp [ 2331.952079] LNet: Removed LNI 192.168.204.42@tcp2 [ 2331.982943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.049660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2332.071939] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2332.735397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2332.748828] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2333.449616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2333.482671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2333.537261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2333.714391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.730724] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.735444] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2334.751684] LNet: Removed LNI 192.168.204.42@tcp [ 2335.783896] LNet: Removed LNI 192.168.204.42@tcp2 [ 2335.840614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.926172] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2335.932512] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2335.941639] LNet: 117777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.451356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2336.653269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.673677] LNet: Removed LNI 192.168.204.42@tcp [ 2337.702642] LNet: Removed LNI 192.168.204.42@tcp2 [ 2337.746864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.797271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2337.829239] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2338.383797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.410331] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2339.050955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2339.084490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2339.146965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2339.501917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.531421] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.542341] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2339.550821] LNet: Removed LNI 192.168.204.42@tcp [ 2339.565335] LNet: Removed LNI 192.168.204.42@tcp2 [ 2339.585668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.662976] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2339.679802] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2340.174788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2340.506378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.535732] LNet: Removed LNI 192.168.204.42@tcp [ 2341.546056] LNet: Removed LNI 192.168.204.42@tcp2 [ 2341.588447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.649413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.683245] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2342.388451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.415066] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2343.379515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2343.414927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2343.448111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2343.513111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2343.926929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.991763] LNet: Removed LNI 192.168.204.42@tcp [ 2346.021199] LNet: Removed LNI 192.168.204.42@tcp2 [ 2346.052619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.126696] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2346.138833] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2346.145110] LNet: 118476:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2346.169830] LNet: 118476:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2346.557635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2353.006789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2354.518835] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:47:09 (1788450429) [ 2354.669991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.717086] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.722403] LNet: 119235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2354.733374] LNet: Removed LNI 192.168.204.42@tcp [ 2355.755447] LNet: Removed LNI 192.168.204.42@tcp2 [ 2355.816396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.142972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.176826] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2356.849782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2356.868888] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2357.929141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.946751] LNetError: 119511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2357.952606] LNetError: 119511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2357.957168] LNet: Removed LNI 10.1.2.3@tcp [ 2357.972738] LNet: Removed LNI 192.168.204.42@tcp [ 2357.999195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.117033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2358.157874] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2358.162810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2358.596470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2358.632455] LNet: Removed LNI 192.168.204.42@tcp [ 2358.644240] LNet: Removed LNI 10.1.2.3@tcp [ 2365.470247] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:47:20 (1788450440) [ 2365.656887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.692758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.755131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.784393] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2366.731656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2366.834040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2366.909812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2366.961246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.005828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.039934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.087680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.129640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.164639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.208701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.242634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.42@tcp [ 2367.443298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2373.825382] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:47:28 (1788450448) [ 2374.005839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.078472] LNet: Removed LNI 192.168.204.42@tcp [ 2375.109135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.223380] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2375.321813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2375.326092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2375.329948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2377.701914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2377.748993] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2378.445925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2378.472551] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.538678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2380.575796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2380.633790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.42@tcp -m PUT -e local_timeout -r 1 [ 2380.682410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2380.802388] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.809486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2381.176458] LNetError: 120917:0:(lib-move.c:4801:lnet_parse()) 192.168.204.42@tcp, src 192.168.204.42@tcp: Dropping PUT (error -108 looking up sender) [ 2382.176501] LNet: Removed LNI 10.1.2.3@tcp [ 2383.212141] LNet: Removed LNI 192.168.204.42@tcp [ 2383.612860] Key type .llcrypt unregistered [ 2383.617190] Key type ._llcrypt unregistered [ 2395.661741] Key type ._llcrypt registered [ 2395.665144] Key type .llcrypt registered [ 2396.872545] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:47:51 (1788450471) [ 2397.355578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.373254] alg: No test for adler32 (adler32-zlib) [ 2398.358955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.695647] LNet: 122224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.210388] Key type .llcrypt unregistered [ 2399.222318] Key type ._llcrypt unregistered [ 2405.797766] Key type ._llcrypt registered [ 2405.799816] Key type .llcrypt registered [ 2406.000930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2406.010515] alg: No test for adler32 (adler32-zlib) [ 2406.881895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2406.944422] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2416.876843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2417.353527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2418.517501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2418.865366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.885748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.912468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.936192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.990033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.039725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.076312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.115672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.133534] LNet: 122671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.182711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2419.254889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.296937] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2419.324458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.382886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.443523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.505526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2420.457588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2420.893102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.941915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2421.006502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2421.058454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2421.127902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.192585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.288442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.376994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.411893] LNet: 122789:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.446949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.499978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.548263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.614606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.658575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.680645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.334895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2422.728551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.774709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.814684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.849941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.911726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.960742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2423.006115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2423.066458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2423.104701] LNet: 122906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2423.136950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2423.188767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.247156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.303392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.367845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.410226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2424.431097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2424.881786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.937816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.993537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2425.062777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2425.102607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2425.135633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2425.182918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2425.234471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2425.262510] LNet: 123023:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2425.293428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2425.335206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2425.378271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2425.404824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2425.433179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2425.467355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.792453] LNet: 123069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.797978] LNetError: 123069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.817170] LNet: Removed LNI 192.168.204.42@tcp [ 2426.304402] Key type .llcrypt unregistered [ 2426.306341] Key type ._llcrypt unregistered [ 2444.815316] Key type ._llcrypt registered [ 2444.817514] Key type .llcrypt registered [ 2446.264356] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:48:40 (1788450520) [ 2446.678525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.689498] alg: No test for adler32 (adler32-zlib) [ 2447.583143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2448.064607] LNet: 123954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2448.725560] Key type .llcrypt unregistered [ 2448.734807] Key type ._llcrypt unregistered [ 2455.469142] Key type ._llcrypt registered [ 2455.473167] Key type .llcrypt registered [ 2455.639591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2455.655571] alg: No test for adler32 (adler32-zlib) [ 2456.547591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2456.570707] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2463.595850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.142@tcp [ 2473.054245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2473.119945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2473.159772] LNet: Added LNI 192.168.204.42@tcp1 [8/256/0/180] [ 2474.313427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2474.947771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.142@tcp [ 2475.039383] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.142@tcp [ 2475.251542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2475.330092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2475.377855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2475.431745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2475.486318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2475.533785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2475.574799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2475.625332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2475.654446] LNet: 124619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2475.677563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2475.716818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2475.755597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2475.798702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2475.842670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2475.880570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2476.659840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2476.880727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2476.906278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2476.939894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2476.968313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2476.991641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2477.014444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2477.044864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2477.077420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2477.092913] LNet: 124736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2477.109734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2477.173330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2477.210892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2477.246990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2477.287539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2477.324762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2477.980500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.142@tcp [ 2478.199717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2478.254541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2478.309146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2478.334966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2478.358157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2478.388483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2478.412852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2478.452937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2478.477330] LNet: 124853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2478.512573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2478.549556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2478.583311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2478.619067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2478.682086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2478.716646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2479.416542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2479.675720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2479.728983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2479.777744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2479.801453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2479.846270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2479.892631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2479.953091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2479.995897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2480.047531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2480.104389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2480.164113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2480.229952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2480.269642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2480.305692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2485.917990] LNet: 125051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.925675] LNetError: 125051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.935991] LNet: Removed LNI 192.168.204.42@tcp [ 2485.958220] LNet: Removed LNI 192.168.204.42@tcp1 [ 2486.576834] Key type .llcrypt unregistered [ 2486.582794] Key type ._llcrypt unregistered [ 2505.157689] Key type ._llcrypt registered [ 2505.160757] Key type .llcrypt registered [ 2506.637780] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:49:41 (1788450581) [ 2507.227700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.245873] alg: No test for adler32 (adler32-zlib) [ 2508.261401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2508.344134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2508.393491] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2509.089872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2509.108348] LNet: Added LNI 192.168.204.42@tcp2 [8/256/0/180] [ 2509.998200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.42@tcp [ 2510.110821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2511.199336] LNet: Removed LNI 192.168.204.42@tcp [ 2518.209120] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:49:52 (1788450592) [ 2519.687312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2521.304957] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:49:56 (1788450596) [ 2528.951555] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:50:03 (1788450603) [ 2536.900630] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:50:11 (1788450611) [ 2544.888480] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:50:19 (1788450619) [ 2553.979494] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:50:28 (1788450628) [ 2561.869345] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:50:36 (1788450636) [ 2569.351528] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:50:43 (1788450643) [ 2576.670223] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:50:51 (1788450651) [ 2584.857621] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:50:59 (1788450659) [ 2592.726608] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:51:07 (1788450667) [ 2600.516502] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:51:15 (1788450675) [ 2608.654948] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:51:23 (1788450683) [ 2616.738376] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:51:31 (1788450691) [ 2624.724606] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:51:39 (1788450699) [ 2634.057919] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:51:48 (1788450708) [ 2643.435813] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:51:57 (1788450717) [ 2651.869828] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:52:06 (1788450726) [ 2659.877981] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:52:14 (1788450734) [ 2660.225478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2660.251896] LNet: 136744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2660.258408] LNetError: 136744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2660.287203] LNet: Removed LNI 192.168.204.42@tcp2 [ 2660.358860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2660.541542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2660.559649] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2662.167052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2662.212963] LNet: 136886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2662.224327] LNetError: 136886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.250326] LNet: Removed LNI 192.168.204.42@tcp [ 2662.283541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2662.376430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2662.402281] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2669.833709] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:52:24 (1788450744) [ 2677.628932] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:52:32 (1788450752) [ 2685.407941] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:52:40 (1788450760) [ 2692.378935] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:52:47 (1788450767) [ 2701.807612] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:52:55 (1788450775) [ 2709.706176] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:53:04 (1788450784) [ 2716.508667] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:53:11 (1788450791) [ 2723.704780] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:53:18 (1788450798) [ 2730.976349] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:53:25 (1788450805) [ 2737.999697] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:53:32 (1788450812) [ 2743.596578] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:53:38 (1788450818) [ 2748.964252] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:53:43 (1788450823) [ 2755.218490] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2477 sec ========== 11:53:50 (1788450830) [ 2755.534939] LNet: 144354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2755.539694] LNetError: 144354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2755.558359] LNet: Removed LNI 192.168.204.42@tcp [ 2756.375867] Key type .llcrypt unregistered [ 2756.381085] Key type ._llcrypt unregistered [ 2769.147098] Key type ._llcrypt registered [ 2769.149952] Key type .llcrypt registered [ 2769.641274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2769.655245] alg: No test for adler32 (adler32-zlib) [ 2770.833248] Lustre: Lustre: Build Version: 2.17.57_103_g02cdcd5 [ 2771.253803] LNet: Added LNI 192.168.204.42@tcp [8/256/0/180] [ 2772.999246] Key type lgssc registered [ 2774.603642] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2843.396395] Lustre: Mounted lustre-client - version 2.17.57_103_g02cdcd5 [ 2849.031560] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2856.339645] Lustre: Unmounted lustre-client [ 2872.734495] Key type lgssc unregistered [ 2873.028213] LNet: 147302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.035460] LNetError: 147302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2873.053749] LNet: Removed LNI 192.168.204.42@tcp [ 2873.779554] Key type .llcrypt unregistered [ 2873.790070] Key type ._llcrypt unregistered