[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 435013654 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2528MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011034] SELinux: Initializing. [ 0.012067] *** VALIDATE selinux *** [ 0.020556] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025911] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028123] *** VALIDATE tmpfs *** [ 0.030290] *** VALIDATE proc *** [ 0.031280] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033332] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034219] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036033] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039798] debug: unmapping init [mem 0xffffffffb0e59000-0xffffffffb0e60fff] [ 0.041216] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043026] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045016] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047015] ... max period: 00007fffffffffff [ 0.048018] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.050317] rcu: Hierarchical SRCU implementation. [ 0.052548] smp: Bringing up secondary CPUs ... [ 0.053521] x86: Booting SMP configuration: [ 0.054022] .... node #0, CPUs: #1 #2 #3 [ 0.057386] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060022] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.134018] node 0 deferred pages initialised in 71ms [ 0.139127] devtmpfs: initialized [ 0.140240] x86/mm: Memory block size: 128MB [ 0.142851] gcov: version magic: 0x41383552 [ 0.144292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153413] pinctrl core: initialized pinctrl subsystem [ 0.155230] [ 0.155898] ************************************************************* [ 0.158079] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160012] ** ** [ 0.163014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.167014] ** This means that this kernel is built to expose internal ** [ 0.170014] ** IOMMU data structures, which may compromise security on ** [ 0.171008] ** your system. ** [ 0.173013] ** ** [ 0.176012] ** If you see this message and you are not debugging the ** [ 0.178012] ** kernel, report this immediately to your vendor! ** [ 0.180016] ** ** [ 0.182024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185013] ************************************************************* [ 0.187751] NET: Registered protocol family 16 [ 0.189455] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193179] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.196068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.200068] cpuidle: using governor menu [ 0.201754] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204520] PCI: Using configuration type 1 for base access [ 0.206131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218045] cryptd: max_cpu_qlen set to 1000 [ 0.220849] ACPI: Added _OSI(Module Device) [ 0.222012] ACPI: Added _OSI(Processor Device) [ 0.223010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.229541] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.235481] ACPI: Interpreter enabled [ 0.237067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237899] ACPI: Using IOAPIC for interrupt routing [ 0.239125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.242392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253226] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.256050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270307] acpiphp: Slot [2] registered [ 0.272130] acpiphp: Slot [5] registered [ 0.274089] acpiphp: Slot [6] registered [ 0.275120] acpiphp: Slot [3] registered [ 0.277131] acpiphp: Slot [4] registered [ 0.278127] acpiphp: Slot [7] registered [ 0.280099] acpiphp: Slot [8] registered [ 0.282099] acpiphp: Slot [9] registered [ 0.283098] acpiphp: Slot [10] registered [ 0.285129] acpiphp: Slot [11] registered [ 0.286102] acpiphp: Slot [12] registered [ 0.288105] acpiphp: Slot [13] registered [ 0.289086] acpiphp: Slot [14] registered [ 0.290078] acpiphp: Slot [15] registered [ 0.292085] acpiphp: Slot [16] registered [ 0.293105] acpiphp: Slot [17] registered [ 0.294273] acpiphp: Slot [18] registered [ 0.296146] acpiphp: Slot [19] registered [ 0.298095] acpiphp: Slot [20] registered [ 0.299086] acpiphp: Slot [21] registered [ 0.302130] acpiphp: Slot [22] registered [ 0.304150] acpiphp: Slot [23] registered [ 0.307120] acpiphp: Slot [24] registered [ 0.309124] acpiphp: Slot [25] registered [ 0.311147] acpiphp: Slot [26] registered [ 0.314131] acpiphp: Slot [27] registered [ 0.315135] acpiphp: Slot [28] registered [ 0.317111] acpiphp: Slot [29] registered [ 0.319179] acpiphp: Slot [30] registered [ 0.321138] acpiphp: Slot [31] registered [ 0.322089] PCI host bridge to bus 0000:00 [ 0.324026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.327029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.330163] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.333026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.336025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.339044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.342399] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.347254] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352492] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.367086] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.370029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378651] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381915] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387966] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.404019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414924] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.458513] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.465017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.506408] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.511416] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.514367] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.517405] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.519298] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.524032] iommu: Default domain type: Passthrough [ 0.525389] SCSI subsystem initialized [ 0.526127] ACPI: bus type USB registered [ 0.527082] usbcore: registered new interface driver usbfs [ 0.529067] usbcore: registered new interface driver hub [ 0.530070] usbcore: registered new device driver usb [ 0.532167] pps_core: LinuxPPS API ver. 1 registered [ 0.533009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.535061] PTP clock support registered [ 0.537118] EDAC MC: Ver: 3.0.0 [ 0.539137] PCI: Using ACPI for IRQ routing [ 0.540828] NetLabel: Initializing [ 0.541014] NetLabel: domain hash size = 128 [ 0.542012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.543094] NetLabel: unlabeled traffic allowed by default [ 0.545145] vgaarb: loaded [ 0.547293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.548015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.552402] clocksource: Switched to clocksource kvm-clock [ 0.659245] VFS: Disk quotas dquot_6.6.0 [ 0.660689] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.663706] *** VALIDATE ramfs *** [ 0.665044] *** VALIDATE hugetlbfs *** [ 0.666918] pnp: PnP ACPI init [ 0.669597] pnp: PnP ACPI: found 6 devices [ 0.703990] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.707746] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.710171] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.712269] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.714803] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.717090] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.720057] NET: Registered protocol family 2 [ 0.722546] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.727524] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.731378] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.739840] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.745334] TCP: Hash tables configured (established 65536 bind 65536) [ 0.750084] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.754082] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.757206] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760428] NET: Registered protocol family 1 [ 0.762903] RPC: Registered named UNIX socket transport module. [ 0.765704] RPC: Registered udp transport module. [ 0.767836] RPC: Registered tcp transport module. [ 0.770017] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.772894] NET: Registered protocol family 44 [ 0.774065] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.775781] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.777297] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.779046] PCI: CLS 0 bytes, default 64 [ 0.780309] Unpacking initramfs... [ 2.183227] debug: unmapping init [mem 0xffff9c5b7cc64000-0xffff9c5b7ffcffff] [ 2.186535] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.188946] software IO TLB: mapped [mem 0x00000000b8c64000-0x00000000bcc64000] (64MB) [ 2.191933] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.209435] Initialise system trusted keyrings [ 3.211843] Key type blacklist registered [ 3.217688] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.238190] zbud: loaded [ 3.245668] *** VALIDATE nfs *** [ 3.247054] *** VALIDATE nfs4 *** [ 3.253669] pstore: using deflate compression [ 3.258496] Platform Keyring initialized [ 3.467492] NET: Registered protocol family 38 [ 3.470058] Key type asymmetric registered [ 3.475214] Asymmetric key parser 'x509' registered [ 3.480078] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.496469] io scheduler mq-deadline registered [ 3.501732] io scheduler kyber registered [ 3.503622] io scheduler bfq registered [ 3.508418] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.513463] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.524714] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.530919] ACPI: Power Button [PWRF] [ 3.546170] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.562813] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.588208] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.643474] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.692082] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.716535] Non-volatile memory driver v1.3 [ 3.721150] Linux agpgart interface v0.103 [ 3.807993] virtio_blk virtio1: [vda] 149496 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.812273] vda: detected capacity change from 0 to 76541952 [ 3.842758] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.848332] vdb: detected capacity change from 0 to 1073741824 [ 3.861767] libphy: Fixed MDIO Bus: probed [ 3.871476] usbcore: registered new interface driver usbserial_generic [ 3.875710] usbserial: USB Serial support registered for generic [ 3.878931] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.885978] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.890867] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.894159] mousedev: PS/2 mouse device common for all mice [ 3.898361] rtc_cmos 00:05: RTC can wake from S4 [ 3.901731] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.905437] rtc_cmos 00:05: registered as rtc0 [ 3.908883] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.912947] intel_pstate: CPU model not supported [ 3.917913] hid: raw HID events driver (C) Jiri Kosina [ 3.921555] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.921994] usbcore: registered new interface driver usbhid [ 3.929791] usbhid: USB HID core driver [ 3.932274] drop_monitor: Initializing network drop monitor service [ 3.935374] Initializing XFRM netlink socket [ 3.936673] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.940591] NET: Registered protocol family 10 [ 3.948682] Segment Routing with IPv6 [ 3.950106] NET: Registered protocol family 17 [ 3.952250] mpls_gso: MPLS GSO support [ 3.958858] RAS: Correctable Errors collector initialized. [ 3.961070] AVX version of gcm_enc/dec engaged. [ 3.962864] AES CTR mode by8 optimization enabled [ 4.081233] sched_clock: Marking stable (4081210319, 0)->(5031810357, -950600038) [ 4.100091] registered taskstats version 1 [ 4.108588] Loading compiled-in X.509 certificates [ 4.113671] zswap: loaded using pool lzo/zbud [ 4.191645] Key type big_key registered [ 4.219552] Key type encrypted registered [ 4.222335] ima: No TPM chip found, activating TPM-bypass! [ 4.227518] ima: Allocated hash algorithm: sha1 [ 4.230275] ima: No architecture policies found [ 4.235993] evm: Initialising EVM extended attributes: [ 4.238889] evm: security.selinux [ 4.240977] evm: security.ima [ 4.242329] evm: security.capability [ 4.243968] evm: HMAC attrs: 0x1 [ 4.247307] rtc_cmos 00:05: setting system clock to 2026-08-22 06:27:17 UTC (1787380037) [ 4.255288] debug: unmapping init [mem 0xffffffffb1e03000-0xffffffffb1ffffff] [ 4.262293] debug: unmapping init [mem 0xffffffffb0b82000-0xffffffffb0e58fff] [ 4.278901] Write protecting the kernel read-only data: 28672k [ 4.288924] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 4.292426] debug: unmapping init [mem 0xffffffffafb14000-0xffffffffafbfffff] [ 4.364419] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.383881] systemd[1]: Detected virtualization kvm. [ 4.390898] systemd[1]: Detected architecture x86-64. [ 4.392790] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.481467] systemd[1]: No hostname configured. [ 4.492269] systemd[1]: Set hostname to . [ 4.501463] random: systemd: uninitialized urandom read (16 bytes read) [ 4.508065] systemd[1]: Initializing machine ID from random generator. [ 4.652672] random: ln: uninitialized urandom read (6 bytes read) [ 4.821739] random: systemd: uninitialized urandom read (16 bytes read) [ 4.826229] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.833902] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.839859] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.271596] device-mapper: uevent: version 1.0.3 [ 6.274960] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.833888] virtio_net virtio0 ens2: renamed from eth0 [ 8.050702] scsi host0: ata_piix [ 8.149783] scsi host1: ata_piix [ 8.154455] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.165100] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.755735] random: crng init done [ 12.759531] random: 7 urandom warning(s) missed due to ratelimiting [ 12.933346] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.656125] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.936857] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.708196] SELinux: Disabled at runtime. [ 17.819973] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.838982] systemd[1]: Detected virtualization kvm. [ 17.842987] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.081884] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.085953] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.093729] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.098414] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.104324] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.123474] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.144717] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ 19.222778] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 20.111768] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.061454] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.113678] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.532444] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.590291] EDAC sbridge: Ver: 1.1.2 [ 24.533166] Key type dns_resolver registered [ 25.023092] NFS: Registering the id_resolver key type [ 25.024741] Key type id_resolver registered [ 25.026918] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy D[ 29.138007] hrtimer: interrupt took 7997225 ns aemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg411-client login: [ 87.507796] libcfs: loading out-of-tree module taints kernel. [ 87.583465] Key type ._llcrypt registered [ 87.587101] Key type .llcrypt registered [ 87.921358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 87.939446] alg: No test for adler32 (adler32-zlib) [ 89.263236] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 90.052241] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 91.839375] Key type lgssc registered [ 93.283060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 207.148574] Lustre: Mounted lustre-client [ 212.518206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.279443] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing check_logdir /tmp/testlogs/ [ 231.695511] Lustre: DEBUG MARKER: oleg411-client.virtnet: executing yml_node [ 232.928869] Lustre: lustre-OST0000-osc-ffff9c5bd88f1000: disconnect after 24s idle [ 235.867938] Lustre: DEBUG MARKER: Client: 2.17.54.116 [ 238.251659] Lustre: DEBUG MARKER: MDS: 2.17.54.116 [ 240.794196] Lustre: DEBUG MARKER: OSS: 2.17.54.116 [ 242.470163] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 02:31:14 EDT 2026 [ 256.663655] Lustre: Unmounted lustre-client [ 290.820237] Key type lgssc unregistered [ 290.992395] LNet: 8386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.996300] LNetError: 8386:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.015778] LNet: Removed LNI 192.168.204.11@tcp [ 291.800171] Key type .llcrypt unregistered [ 291.804090] Key type ._llcrypt unregistered [ 315.351979] Key type ._llcrypt registered [ 315.358984] Key type .llcrypt registered [ 315.405082] LNet: 8965:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 315.405082] mask usage: [+|-] ... [ 315.651105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 315.684903] alg: No test for adler32 (adler32-zlib) [ 316.571755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 316.664665] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 316.696417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 320.763511] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 320.870992] LNet: 9261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.881683] LNetError: 9261:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 320.912868] LNet: Removed LNI 192.168.204.11@tcp [ 321.508218] Key type .llcrypt unregistered [ 321.509645] Key type ._llcrypt unregistered [ 326.950165] Key type ._llcrypt registered [ 326.954870] Key type .llcrypt registered [ 327.867760] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:32:40 (1787380360) [ 327.972445] LNet: 9639:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 327.972445] mask usage: [+|-] ... [ 328.085093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.096236] alg: No test for adler32 (adler32-zlib) [ 328.965684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.030734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.047562] LNet: 9680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.066920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.135332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.153470] LNet: 9687:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 334.881265] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:32:46 (1787380366) [ 335.018795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.044076] LNet: 10259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.062793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.159810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 341.069821] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:32:53 (1787380373) [ 341.281295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 341.301632] LNet: 10850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 341.313980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 341.411777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 347.902330] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:32:59 (1787380379) [ 348.079073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.109583] LNet: 11432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.126622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.205315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 355.246343] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:33:06 (1787380386) [ 355.423834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.486081] LNet: 12020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.549893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.641858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 356.004675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.395569] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:33:14 (1787380394) [ 362.534925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.603313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.692370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.053234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.394855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 369.551956] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:33:21 (1787380401) [ 369.706716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.728843] LNet: 13208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.733640] LNet: 13208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.755142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.815728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 375.618147] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:33:27 (1787380407) [ 375.983880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.029794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.104643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 376.131546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.487411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 376.524691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.915641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 376.971530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.333384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 377.371270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.869116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 377.907809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.303830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 378.343663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.694269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 378.720560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.104682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 379.158589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.607247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 379.630841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.986113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 380.009538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.344921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 380.377342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.838551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 380.884211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 381.354749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.388205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.940975] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:33:38 (1787380418) [ 387.122712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.148728] LNet: 14473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.154023] LNet: 14473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 387.171801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.241915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 387.291342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 393.089530] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:33:45 (1787380425) [ 393.402152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.465498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.547659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 393.591849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 399.561031] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:33:51 (1787380431) [ 399.635476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.676503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.775728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 399.816778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 405.551725] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:33:57 (1787380437) [ 405.715474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.755864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.832575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 405.873529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 411.659394] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:34:03 (1787380443) [ 411.868407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.896361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.955514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 411.993839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 418.221967] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:34:10 (1787380450) [ 418.376060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.418441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.523155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 418.571440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 424.664609] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:34:16 (1787380456) [ 424.897599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.920663] LNet: 17992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.927492] LNet: 17992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 424.958300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.066876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.610316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.600578] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:34:23 (1787380463) [ 431.690705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.755112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.018396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.274625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 437.878953] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:34:29 (1787380469) [ 438.020411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.083664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.191790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 438.399491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.189138] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:34:36 (1787380476) [ 444.397218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.440673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.010259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.194308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.544633] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:34:44 (1787380484) [ 452.634539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.696711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.807755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.890768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 459.137399] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:34:51 (1787380491) [ 459.441383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.474613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.549574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.597524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 466.969125] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:34:58 (1787380498) [ 467.155639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.190500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.239473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 472.955722] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:35:04 (1787380504) [ 473.053574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.100310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.187676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 478.947720] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:35:10 (1787380510) [ 479.222992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.250307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.327358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 479.757556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 485.477618] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:35:17 (1787380517) [ 485.684210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.717259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.830613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 485.886631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 491.645303] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:35:23 (1787380523) [ 491.785429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.825162] LNet: 24012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 491.830018] LNet: 24012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 491.871442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.965961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 491.992464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 497.394460] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:35:29 (1787380529) [ 497.533783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.562628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.620194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 497.643957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 502.938906] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:35:35 (1787380535) [ 503.054136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.115770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.189544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 503.216360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 507.925141] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:35:40 (1787380540) [ 507.999433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.050051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.115669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 508.166606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 508.185800] LNetError: 25788:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 508.226938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 508.256878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 513.411450] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:35:45 (1787380545) [ 513.512394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.564352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.681958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Xdu59jqLul [ 513.767151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Xdu59jqLul [ 513.808325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Xdu59jqLul [ 518.967840] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:35:51 (1787380551) [ 519.186727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.216316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.247887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 519.275349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 519.329806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 524.477619] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:35:56 (1787380556) [ 524.769135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.822371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.907065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 527.229680] LNet: Message forwarding will be enabled in 2s [ 530.336034] LNet: Message forwarding enabled [ 530.671157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 530.684479] LNet: Message forwarding will stop in 2s to 6s [ 535.105371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 535.359563] LNet: Message forwarding disabled [ 536.891599] LNet: Message forwarding will be enabled in 2s [ 540.512431] LNet: Message forwarding enabled [ 542.546129] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:36:14 (1787380574) [ 542.649038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.669187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.736542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 548.365869] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:36:20 (1787380580) [ 548.430677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.470271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.512270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 548.560333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 548.584854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 548.611524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 548.645210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 548.676697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 548.708646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 548.748986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 548.785750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 548.815244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 548.839324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 548.872639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 548.904338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 548.928923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 548.954561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 548.981855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 549.021771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 549.050628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 549.075978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 549.095942] LNetError: 28858:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 554.176714] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:36:26 (1787380586) [ 554.295550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.337233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.417433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 560.255804] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:36:32 (1787380592) [ 561.802846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg411-server [ 563.602291] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:36:35 (1787380595) [ 563.684559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.730715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.790497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 563.821260] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 565.221582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[12-16]@tcp [ 565.255796] LNet: 30573:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 565.310907] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 565.335635] LNetError: 30361:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.14@tcp has gone from up to down [ 566.308927] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 566.321888] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 567.330260] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 567.341227] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 569.377655] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 569.391963] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 570.985636] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:36:42 (1787380602) [ 572.577930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg411-server [ 573.473496] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 573.491938] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 573.907481] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:36:46 (1787380606) [ 574.092316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.146807] LNetError: 31508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.172450] LNet: Removed LNI 192.168.204.11@tcp [ 574.248719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.315395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 574.360813] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 575.013795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 575.034749] LNet: 31625:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 575.041561] LNet: 31625:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 575.057144] LNetError: 31512:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.14@tcp has gone from up to down [ 575.064173] LNetError: 31512:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 575.071301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[12-17/2]@tcp [ 579.385664] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:36:51 (1787380611) [ 579.468378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.484304] LNetError: 32211:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 579.503414] LNet: Removed LNI 192.168.204.11@tcp [ 579.531435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.605551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 579.644800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.670210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.732613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.768239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.791481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.824075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 579.878869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.906961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.959972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.001576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.029324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.067949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 580.116807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.147567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.191782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.247677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.269740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.307523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 580.366076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.394465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.436574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.488338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.525909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.569751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 580.627932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.648470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.695616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 580.748263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.787617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.821839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 580.871253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.906478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.963312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.167500] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:36:57 (1787380617) [ 586.428140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg411-server [ 587.658601] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:36:59 (1787380619) [ 588.985740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg411-server [ 590.271714] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:37:02 (1787380622) [ 590.897473] Key type .llcrypt unregistered [ 590.899368] Key type ._llcrypt unregistered [ 595.614646] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 595.764121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 595.770365] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 595.775482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 598.133936] Key type ._llcrypt registered [ 598.137366] Key type .llcrypt registered [ 598.166298] LNet: 33897:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 598.166298] mask usage: [+|-] ... [ 598.286233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 598.304921] alg: No test for adler32 (adler32-zlib) [ 599.184448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.233878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.262521] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 599.794302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 599.816801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 601.215932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 601.236021] LNet: Removed LNI 10.1.2.3@tcp [ 601.326657] LNet: 34338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.333659] LNetError: 34338:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.349613] LNet: Removed LNI 192.168.204.11@tcp [ 601.782493] Key type .llcrypt unregistered [ 601.784183] Key type ._llcrypt unregistered [ 609.241233] Key type ._llcrypt registered [ 609.243327] Key type .llcrypt registered [ 610.210430] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:37:22 (1787380642) [ 610.612439] Key type .llcrypt unregistered [ 610.614583] Key type ._llcrypt unregistered [ 614.934072] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 615.058815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 615.064443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 615.070626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 617.588287] Key type ._llcrypt registered [ 617.590363] Key type .llcrypt registered [ 617.613994] LNet: 35372:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 617.613994] mask usage: [+|-] ... [ 617.745744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.761719] alg: No test for adler32 (adler32-zlib) [ 618.611777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.658490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.676888] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 619.294464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 619.310044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 621.208624] LNet: 35829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.214844] LNetError: 35829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 621.219762] LNet: Removed LNI 10.1.2.3@tcp [ 621.229903] LNet: Removed LNI 192.168.204.11@tcp [ 621.643697] Key type .llcrypt unregistered [ 621.647322] Key type ._llcrypt unregistered [ 629.393992] Key type ._llcrypt registered [ 629.396272] Key type .llcrypt registered [ 630.377419] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:37:42 (1787380662) [ 630.771520] Key type .llcrypt unregistered [ 630.775424] Key type ._llcrypt unregistered [ 635.110642] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 635.226235] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 635.231189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 635.239602] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 637.762506] Key type ._llcrypt registered [ 637.767975] Key type .llcrypt registered [ 637.801384] LNet: 36864:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 637.801384] mask usage: [+|-] ... [ 637.918587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.930497] alg: No test for adler32 (adler32-zlib) [ 638.830442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.872982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 638.905079] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 640.358342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 640.379539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.011980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 642.037551] LNet: Removed LNI 10.1.2.3@tcp [ 642.054327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 642.093806] LNet: Removed LNI 10.1.2.31@tcp [ 642.118253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 642.146383] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.730815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 642.748270] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 643.438033] LNet: 37746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.444500] LNetError: 37746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 643.450324] LNet: Removed LNI 10.1.2.31@tcp [ 643.460927] LNet: Removed LNI 10.1.2.3@tcp [ 643.942488] Key type .llcrypt unregistered [ 643.944756] Key type ._llcrypt unregistered [ 653.990471] Key type ._llcrypt registered [ 653.992633] Key type .llcrypt registered [ 655.012658] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:38:07 (1787380687) [ 655.189416] LNet: 38539:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 655.189416] mask usage: [+|-] ... [ 655.310458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 655.326878] alg: No test for adler32 (adler32-zlib) [ 656.200705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.249090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.266334] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 656.915721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.945147] LNet: 38699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.955393] LNetError: 38699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.967420] LNet: Removed LNI 192.168.204.11@tcp [ 656.988282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.047996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.11@tcp [ 657.072728] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 657.731432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.11@tcp [ 657.752882] LNet: Removed LNI 192.168.204.11@tcp [ 657.811276] LNet: 38834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.356574] Key type .llcrypt unregistered [ 658.360518] Key type ._llcrypt unregistered [ 666.593502] Key type ._llcrypt registered [ 666.596606] Key type .llcrypt registered [ 667.712789] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:38:19 (1787380699) [ 669.128397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg411-server [ 670.462304] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:38:22 (1787380702) [ 670.902582] Key type .llcrypt unregistered [ 670.907422] Key type ._llcrypt unregistered [ 676.649564] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 676.803947] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 676.808553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 676.812588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 679.269772] Key type ._llcrypt registered [ 679.271917] Key type .llcrypt registered [ 679.306490] LNet: 40227:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 679.306490] mask usage: [+|-] ... [ 679.422322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.432957] alg: No test for adler32 (adler32-zlib) [ 680.310803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.359670] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 680.873512] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 682.593667] LNet: 40686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.605692] LNetError: 40686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.611791] LNet: Removed LNI 192.168.204.11@tcp1 [ 682.622733] LNet: Removed LNI 10.1.2.3@tcp2 [ 683.151230] Key type .llcrypt unregistered [ 683.154577] Key type ._llcrypt unregistered [ 694.167742] Key type ._llcrypt registered [ 694.170716] Key type .llcrypt registered [ 695.832719] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:38:47 (1787380727) [ 696.092268] LNet: 41487:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 696.092268] mask usage: [+|-] ... [ 696.313949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.368798] alg: No test for adler32 (adler32-zlib) [ 697.441852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.499586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.522106] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 698.492113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.518232] LNet: 41655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.525993] LNetError: 41655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.546775] LNet: Removed LNI 192.168.204.11@tcp [ 698.564874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.656854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 698.701106] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 706.034890] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:38:57 (1787380737) [ 706.167789] LNet: 42241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.180717] LNetError: 42241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.212580] LNet: Removed LNI 192.168.204.11@tcp [ 706.997502] Key type .llcrypt unregistered [ 707.000614] Key type ._llcrypt unregistered [ 713.135196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 713.295982] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 713.299087] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 713.303963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.895752] Key type ._llcrypt registered [ 715.899763] Key type .llcrypt registered [ 715.940544] LNet: 42518:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 715.940544] mask usage: [+|-] ... [ 716.070581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.089328] alg: No test for adler32 (adler32-zlib) [ 716.975854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.059612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.106428] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 717.946958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.963753] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.513818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.527840] LNet: 42985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.531912] LNetError: 42985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 719.540882] LNet: Removed LNI 10.1.2.3@tcp [ 719.556874] LNet: Removed LNI 192.168.204.11@tcp [ 719.578428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.616166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 719.633885] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 719.638685] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.936705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.975423] LNet: Removed LNI 10.1.2.3@tcp [ 720.000574] LNet: Removed LNI 192.168.204.11@tcp [ 720.023300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.117473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 720.147281] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 720.158611] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.605798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 720.623489] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 721.470153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.484514] LNet: 43174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.488294] LNet: 43174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 721.493149] LNetError: 43174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 721.496561] LNetError: 43174:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 721.501061] LNet: Removed LNI 10.1.2.3@tcp [ 721.502550] LNet: Removed LNI 192.168.204.11@tcp [ 721.511937] LNet: Removed LNI 192.168.204.11@tcp2 [ 721.532310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.575741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 721.602067] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 721.607584] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 721.614761] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 726.930812] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:39:19 (1787380759) [ 727.015746] LNet: 43765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.023804] LNetError: 43765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 727.030071] LNetError: 43765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 727.034837] LNet: Removed LNI 10.1.2.3@tcp [ 727.036893] LNet: Removed LNI 192.168.204.11@tcp [ 727.051284] LNet: Removed LNI 192.168.204.11@tcp2 [ 727.633285] Key type .llcrypt unregistered [ 727.636802] Key type ._llcrypt unregistered [ 732.622160] Key type ._llcrypt registered [ 732.626268] Key type .llcrypt registered [ 732.661890] LNet: 43996:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 732.661890] mask usage: [+|-] ... [ 732.803510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.817453] alg: No test for adler32 (adler32-zlib) [ 733.722260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.759381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.790715] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 734.430585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.452657] LNet: 44165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.457295] LNetError: 44165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.473542] LNet: Removed LNI 192.168.204.11@tcp [ 734.494775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.596372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 734.636322] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 734.966489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.984111] LNet: 44195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.988552] LNetError: 44195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.001227] LNet: Removed LNI 192.168.204.11@tcp [ 735.016398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.058724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 735.077576] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 735.551470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.577969] LNet: Removed LNI 192.168.204.11@tcp [ 735.622065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.692950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 735.714804] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 736.142240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 736.276875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 736.284511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 736.291387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 738.745987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.771951] LNet: 44303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.778062] LNet: 44303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 738.788522] LNetError: 44303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.796135] LNetError: 44303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 738.815450] LNet: Removed LNI 192.168.204.11@tcp [ 738.827223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.885396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 738.918496] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 745.538246] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:39:37 (1787380777) [ 745.717094] LNet: 44900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.725688] LNetError: 44900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.738974] LNet: Removed LNI 192.168.204.11@tcp [ 746.323447] Key type .llcrypt unregistered [ 746.328030] Key type ._llcrypt unregistered [ 751.408881] Key type ._llcrypt registered [ 751.411844] Key type .llcrypt registered [ 751.446784] LNet: 45125:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 751.446784] mask usage: [+|-] ... [ 751.595410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 751.609528] alg: No test for adler32 (adler32-zlib) [ 752.569519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.633599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 752.654100] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 753.530530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.548613] LNet: 45299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.556843] LNetError: 45299:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.570670] LNet: Removed LNI 192.168.204.11@tcp [ 753.601891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.657216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 753.697313] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 758.986374] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:39:51 (1787380791) [ 759.089102] LNet: 45882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.093752] LNetError: 45882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.120594] LNet: Removed LNI 192.168.204.11@tcp [ 759.598926] Key type .llcrypt unregistered [ 759.601062] Key type ._llcrypt unregistered [ 764.586453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.691574] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 764.693793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 764.695806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.185327] Key type ._llcrypt registered [ 767.187861] Key type .llcrypt registered [ 767.274948] LNet: 46164:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 767.274948] mask usage: [+|-] ... [ 767.421370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.435947] alg: No test for adler32 (adler32-zlib) [ 768.367946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.427188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.459293] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 769.548544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.573269] LNet: 46337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.582887] LNetError: 46337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.615736] LNet: Removed LNI 192.168.204.11@tcp [ 769.633757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.716105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 769.759473] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 777.375857] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:40:08 (1787380808) [ 777.482862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.509812] LNet: 46932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.515712] LNetError: 46932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.537053] LNet: Removed LNI 192.168.204.11@tcp [ 777.577839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.641851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 783.319799] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:40:15 (1787380815) [ 783.481931] LNet: 47518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.104356] Key type .llcrypt unregistered [ 784.110885] Key type ._llcrypt unregistered [ 790.764090] Key type ._llcrypt registered [ 790.767604] Key type .llcrypt registered [ 790.799921] LNet: 47743:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 790.799921] mask usage: [+|-] ... [ 790.911353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.920862] alg: No test for adler32 (adler32-zlib) [ 791.816302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.873874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.907043] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 792.720902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.756686] LNet: 47915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.774474] LNetError: 47915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.793960] LNet: Removed LNI 192.168.204.11@tcp [ 792.808736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.890411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 792.924990] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 798.751852] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:40:30 (1787380830) [ 798.936931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.958925] LNet: 48508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.968662] LNetError: 48508:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.978168] LNet: Removed LNI 192.168.204.11@tcp [ 798.997174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.075374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 804.120770] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:40:36 (1787380836) [ 805.523747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 806.820435] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:40:39 (1787380839) [ 808.230073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 809.777941] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:40:41 (1787380841) [ 811.370531] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 813.113570] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:40:45 (1787380845) [ 814.893679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 816.306651] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:40:48 (1787380848) [ 817.765153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 819.449593] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:40:51 (1787380851) [ 819.578821] LNet: 50551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.452312] Key type .llcrypt unregistered [ 820.455819] Key type ._llcrypt unregistered [ 825.868238] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.039412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.048838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.065049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.518729] Key type ._llcrypt registered [ 828.521475] Key type .llcrypt registered [ 828.562493] LNet: 50841:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 828.562493] mask usage: [+|-] ... [ 828.722441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.741591] alg: No test for adler32 (adler32-zlib) [ 829.685741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.821749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 829.842485] LNet: Added LNI 192.168.204.11@tcp [8/512/0/180] [ 829.849977] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 829.856257] LNet: 50880:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 829.950027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.987601] LNet: 50899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.001575] LNetError: 50899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.016492] LNet: Removed LNI 10.1.2.3@tcp [ 830.019687] LNetError: 50899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.035744] LNet: Removed LNI 192.168.204.11@tcp [ 830.101719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.212548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 830.237511] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 830.298759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.319449] LNet: 50924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.325860] LNetError: 50924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.342784] LNet: Removed LNI 192.168.204.11@tcp [ 830.363241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.420603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 830.441613] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 830.449626] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.454240] LNet: 50932:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 830.551571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.572236] LNet: 50950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.578794] LNetError: 50950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.585754] LNet: Removed LNI 10.1.2.3@tcp [ 830.591606] LNetError: 50950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.608404] LNet: Removed LNI 192.168.204.11@tcp [ 830.630275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.704293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 830.735263] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 830.741141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 830.743558] LNet: 50958:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 830.939814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.960656] LNet: 50984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.966665] LNetError: 50984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 830.975359] LNet: Removed LNI 10.1.2.3@tcp [ 830.978650] LNetError: 50984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.996868] LNet: Removed LNI 192.168.204.11@tcp [ 831.028593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.109449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 831.129642] LNet: Added LNI 192.168.204.11@tcp [8/256/0/182] [ 831.135159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 831.238164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.260470] LNet: 51012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.273769] LNetError: 51012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.286860] LNet: Removed LNI 10.1.2.3@tcp [ 831.292210] LNetError: 51012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.308607] LNet: Removed LNI 192.168.204.11@tcp [ 831.333189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.459988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 831.492598] LNet: Added LNI 192.168.204.11@tcp [8/256/0/182] [ 831.498858] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 831.500601] LNet: 51020:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 831.576322] LNet: Removed LNI 10.1.2.3@tcp [ 831.590413] LNet: Removed LNI 192.168.204.11@tcp [ 832.267603] Key type .llcrypt unregistered [ 832.271784] Key type ._llcrypt unregistered [ 842.439305] Key type ._llcrypt registered [ 842.441941] Key type .llcrypt registered [ 844.128472] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:41:15 (1787380875) [ 844.316706] LNet: 51833:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 844.316706] mask usage: [+|-] ... [ 844.453732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 844.467133] alg: No test for adler32 (adler32-zlib) [ 845.401203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.468933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 845.517037] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 846.405492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.11 [ 846.435949] LNet: 51995:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 846.441167] LNet: Removed LNI 192.168.204.11@tcp [ 851.365868] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:41:23 (1787380883) [ 851.480367] LNet: 52557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.486836] LNetError: 52557:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.500301] LNet: Removed LNI 192.168.204.11@tcp [ 852.010444] Key type .llcrypt unregistered [ 852.014325] Key type ._llcrypt unregistered [ 857.334792] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 857.635706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 860.354854] Key type ._llcrypt registered [ 860.359764] Key type .llcrypt registered [ 860.390836] LNet: 52846:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 860.390836] mask usage: [+|-] ... [ 860.549044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.590286] alg: No test for adler32 (adler32-zlib) [ 861.794657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 867.060587] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:41:39 (1787380899) [ 867.134807] LNet: 53468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.154340] LNetError: 53468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.177311] LNet: Removed LNI 10.1.2.3@tcp [ 867.824659] Key type .llcrypt unregistered [ 867.829927] Key type ._llcrypt unregistered [ 873.823264] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 874.043486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 874.047906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 874.052681] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 876.898955] Key type ._llcrypt registered [ 876.905527] Key type .llcrypt registered [ 876.956829] LNet: 53751:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 876.956829] mask usage: [+|-] ... [ 877.080435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.093572] alg: No test for adler32 (adler32-zlib) [ 878.226897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.470054] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:41:57 (1787380917) [ 887.140305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 889.136353] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:42:00 (1787380920) [ 889.254721] LNet: 54654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.265147] LNetError: 54654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 889.311127] LNet: Removed LNI 10.1.2.3@tcp [ 889.984125] Key type .llcrypt unregistered [ 889.986797] Key type ._llcrypt unregistered [ 896.788466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 897.038970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 897.042147] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 897.052183] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 898.481665] Key type ._llcrypt registered [ 898.483241] Key type .llcrypt registered [ 898.507993] LNet: 54927:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 898.507993] mask usage: [+|-] ... [ 898.623359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.633332] alg: No test for adler32 (adler32-zlib) [ 899.630684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.700783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 908.363901] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:42:20 (1787380940) [ 908.873602] LNet: 55885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.880306] LNetError: 55885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 908.901520] LNet: Removed LNI 10.1.2.3@tcp [ 909.557239] Key type .llcrypt unregistered [ 909.559062] Key type ._llcrypt unregistered [ 915.567463] Key type ._llcrypt registered [ 915.569116] Key type .llcrypt registered [ 915.643807] LNet: 56108:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 915.643807] mask usage: [+|-] ... [ 915.880625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 915.908140] alg: No test for adler32 (adler32-zlib) [ 916.839556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 916.862100] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 925.612645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 926.224922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 926.254645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 926.299989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 926.316400] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 926.574665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.627939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 926.702910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.778564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.843618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.900617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.923208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.946304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.958281] LNet: 56322:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.970692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 927.000149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 927.028698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 927.049373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 927.082406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.109320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.132311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 927.536189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 927.584705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 927.647406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 927.670908] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 927.903711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 927.927308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 927.950975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 927.975148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 927.995405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.028065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.048491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.076957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.088103] LNet: 56432:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.108141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.133532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 928.161431] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 928.161775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 928.214078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 928.257191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 928.287564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 928.322808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 928.912843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 928.950308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 928.986578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 929.022877] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 929.038095] LNet: Skipped 1 previous similar message [ 929.323513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.364640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.398253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.429147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.463609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.492862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.544761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.586950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.619445] LNet: 56541:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.657640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.718799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.749624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.788270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.820278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.856462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.893225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 930.512073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 930.539715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 930.577234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 930.891712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.946264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 931.007297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 931.051537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 931.081902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 931.136131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 931.179487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 931.240386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 931.326447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 931.372728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 931.407453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 931.460758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 931.500980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 931.545505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 931.588071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 932.003084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 932.051583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 932.088842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 932.109480] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 932.113426] LNet: Skipped 1 previous similar message [ 932.338754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.367301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 932.393779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 932.423732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 932.454740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 932.483870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.513190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.543993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.558253] LNet: 56759:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 932.563412] LNet: 56759:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 932.578169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.605741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.644784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.671505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 932.705787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.738213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.766257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 933.286988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 933.314295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 933.352726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 933.529903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 933.560953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 933.589705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 933.613299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 933.646952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 933.685790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 933.745382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 933.806526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 933.875561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 933.941783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 933.988289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 934.058676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 934.142336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.192995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.229774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 934.442572] LNet: 56910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.451401] LNetError: 56910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.470837] LNet: Removed LNI 192.168.204.11@tcp [ 934.920707] Key type .llcrypt unregistered [ 934.926526] Key type ._llcrypt unregistered [ 950.460719] Key type ._llcrypt registered [ 950.462364] Key type .llcrypt registered [ 951.580248] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:43:03 (1787380983) [ 952.461513] Key type .llcrypt unregistered [ 952.467679] Key type ._llcrypt unregistered [ 957.609662] Key type ._llcrypt registered [ 957.611804] Key type .llcrypt registered [ 957.656639] LNet: 57946:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 957.656639] mask usage: [+|-] ... [ 957.879635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.905686] alg: No test for adler32 (adler32-zlib) [ 958.800670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 958.835378] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 966.263971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 976.863260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.892650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 976.919644] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 978.340071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 978.390701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 978.446212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 978.493225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 978.555730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 1 [ 978.590687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 1 [ 978.645880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 1 [ 978.705164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 1 [ 978.767420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 978.804750] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 978.826306] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 979.081220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.115733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 979.161755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 979.192037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 979.220942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 979.247935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 979.277439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 979.306317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 979.323510] LNet: 58375:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 979.340670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 979.369993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 979.404379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 979.434359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.467460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.499520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 979.543124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 980.157914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 980.223602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 980.272859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 980.312371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 980.354108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 1 [ 980.403208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 1 [ 980.451069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 1 [ 980.500337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 1 [ 980.543255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 980.565223] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 980.569414] LNet: Skipped 2 previous similar messages [ 980.816525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.847495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 980.872379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.910451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.935974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.967356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.995271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.019881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.051333] LNet: 58496:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.078584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.111757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.160926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.207331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.241382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.303683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.366406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 981.907772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 981.941581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 981.975876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 982.025323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 982.070547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 1 [ 982.100298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 1 [ 982.147561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 1 [ 982.183833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 1 [ 982.226989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 982.245412] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 982.250299] LNet: Skipped 2 previous similar messages [ 982.463874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 982.502134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 982.533339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 982.559436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 982.590704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 982.630697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 982.662856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 982.694905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 982.712695] LNet: 58618:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.735548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 982.766329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 982.797514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 982.826528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 982.860862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.888986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.934528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 983.462607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 983.491273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 983.528297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 983.557593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 983.602725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 1 [ 983.646426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 1 [ 983.699097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 1 [ 983.751682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 1 [ 983.809617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 984.087553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.124682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 984.167474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 984.199336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 984.245525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 984.286831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 984.317395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 984.342507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 984.371609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 984.398353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 984.426172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 984.466604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 984.504864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 984.541799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 984.596703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 985.068024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 985.102340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 985.136646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 985.162570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 985.204519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 1 [ 985.248276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 1 [ 985.293157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 1 [ 985.330536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 1 [ 985.374678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 985.392314] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 985.397748] LNet: Skipped 5 previous similar messages [ 985.611308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 985.639228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 985.669619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.691987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.718202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.741403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.763836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.788507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.802507] LNet: 58860:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.807273] LNet: 58860:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 985.820659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.848098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.875450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.911728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.948972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.979238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 986.005255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 986.512331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 986.550571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 986.589683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 986.642482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 986.697695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_error -r 1 [ 986.753587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_error -r 1 [ 986.800782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_error -r 1 [ 986.844502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_error -r 1 [ 986.883588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 987.102623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.163231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 987.231366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 987.261070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 987.306665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 987.346744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 987.387659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 987.440514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.485241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.517117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.548228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.574746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.608503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.646151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.688512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 992.738835] LNet: 59057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.748044] LNetError: 59057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.757391] LNet: Removed LNI 192.168.204.11@tcp [ 993.829927] LNet: Removed LNI 192.168.204.11@tcp1 [ 994.286163] Key type .llcrypt unregistered [ 994.291404] Key type ._llcrypt unregistered [ 1009.537742] Key type ._llcrypt registered [ 1009.540877] Key type .llcrypt registered [ 1010.820898] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:44:02 (1787381042) [ 1011.486849] Key type .llcrypt unregistered [ 1011.488755] Key type ._llcrypt unregistered [ 1017.562479] Key type ._llcrypt registered [ 1017.568773] Key type .llcrypt registered [ 1017.618230] LNet: 60092:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1017.618230] mask usage: [+|-] ... [ 1017.812517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.840892] alg: No test for adler32 (adler32-zlib) [ 1018.818454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1018.849947] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1026.107177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1026.758845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1026.793568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1026.828777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1026.848714] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1027.112271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.138594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1027.171413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.195727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.224270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1027.250754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1027.291971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1027.324833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.346573] LNet: 60306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.381052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.442762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.497812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.542830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.581233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.615229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.643595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1028.179626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1028.219185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1028.256647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1028.275495] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1028.540247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.599611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.635418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.663896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.696609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.736242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.783895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.824291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.840764] LNet: 60415:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.859816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.892375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.935723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.967541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.991759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.057173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.105844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.672354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1029.712859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1029.750647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1029.768684] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1029.939702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.963048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1029.984746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.018603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.047825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.104941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.128666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.160129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.186432] LNet: 60525:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.205895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.235650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.266777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.303535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.344415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.394858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.434794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.639642] LNet: 60566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.646683] LNetError: 60566:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.663737] LNet: Removed LNI 192.168.204.11@tcp [ 1031.134280] Key type .llcrypt unregistered [ 1031.137758] Key type ._llcrypt unregistered [ 1047.427983] Key type ._llcrypt registered [ 1047.430466] Key type .llcrypt registered [ 1048.597332] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:44:40 (1787381080) [ 1049.503343] Key type .llcrypt unregistered [ 1049.505527] Key type ._llcrypt unregistered [ 1056.072176] Key type ._llcrypt registered [ 1056.078305] Key type .llcrypt registered [ 1056.142720] LNet: 61603:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1056.142720] mask usage: [+|-] ... [ 1056.305732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.328620] alg: No test for adler32 (adler32-zlib) [ 1057.268685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1057.296128] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1065.477519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1073.943902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.982348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1074.007805] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1075.024495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1075.057726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1075.092436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1075.115505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1075.141210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 1 [ 1075.165941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 1 [ 1075.206860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 1 [ 1075.256618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 1 [ 1075.285480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1075.305202] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1075.312917] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1075.533831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.584921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.617395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.647780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.691575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.738064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.787672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.841082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.860846] LNet: 62032:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.880405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.916592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.952624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.000806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.036365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.070577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.095726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1076.121509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.575960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1076.619783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1076.655743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1076.701968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1076.730677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_error -r 1 [ 1076.757795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_error -r 1 [ 1076.786343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_error -r 1 [ 1076.825279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_error -r 1 [ 1076.848280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1076.866602] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1076.876659] LNet: Skipped 2 previous similar messages [ 1077.060282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.095666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1077.130598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.158668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.187463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.223986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.268955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.315605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.330842] LNet: 62155:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.368507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.415866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.454384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.482429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.522874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.556877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.593526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1077.634431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.146846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1078.181163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1078.218889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1078.244944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1078.280344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_timeout -r 1 [ 1078.311857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_timeout -r 1 [ 1078.347985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_timeout -r 1 [ 1078.379570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_timeout -r 1 [ 1078.419962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.111@tcp [ 1078.447402] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1078.450201] LNet: Skipped 1 previous similar message [ 1078.648841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.679652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1078.714223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.747062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.783950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.822125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.848850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.874564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.888473] LNet: 62278:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.902770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.924746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.953823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.017081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.070692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.101913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.128927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1079.158854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.055417] LNet: 62356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.070759] LNetError: 62356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.082223] LNet: Removed LNI 192.168.204.11@tcp [ 1083.102810] LNet: Removed LNI 192.168.204.11@tcp1 [ 1083.573413] Key type .llcrypt unregistered [ 1083.579317] Key type ._llcrypt unregistered [ 1099.049732] Key type ._llcrypt registered [ 1099.051501] Key type .llcrypt registered [ 1100.442555] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:45:32 (1787381132) [ 1102.109563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1103.679571] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:45:35 (1787381135) [ 1104.586340] Key type .llcrypt unregistered [ 1104.588581] Key type ._llcrypt unregistered [ 1111.641661] Key type ._llcrypt registered [ 1111.644624] Key type .llcrypt registered [ 1111.677665] LNet: 63696:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1111.677665] mask usage: [+|-] ... [ 1111.909554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1111.934985] alg: No test for adler32 (adler32-zlib) [ 1112.843048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.895874] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1122.206852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1123.330643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1123.371939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1123.425435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1123.465034] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1123.810671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.868057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1123.916556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.961342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.018127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.062230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.098200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.142932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.182846] LNet: 63911:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.191477] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1124.195535] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1124.247590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.364583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.432861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.531456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.608905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.722819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.078447] LNet: 63950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.101839] LNetError: 63950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.129849] LNet: Removed LNI 192.168.204.11@tcp [ 1125.985512] Key type .llcrypt unregistered [ 1125.992626] Key type ._llcrypt unregistered [ 1146.232755] Key type ._llcrypt registered [ 1146.236044] Key type .llcrypt registered [ 1146.314570] LNet: 64400:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1146.314570] mask usage: [+|-] ... [ 1146.510859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1146.529717] alg: No test for adler32 (adler32-zlib) [ 1147.532960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1147.566797] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1155.599492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1165.563612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.612797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.656380] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1167.064853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1167.099617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1167.145382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1167.175416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1167.208558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e network_timeout -r 1 [ 1167.243553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e network_timeout -r 1 [ 1167.288707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e network_timeout -r 1 [ 1167.334095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e network_timeout -r 1 [ 1167.376895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1167.397509] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1167.632318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.667616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1167.697654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.733541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.763052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.788270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.823410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.848510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.865542] LNet: 64829:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.886620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.912378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.939716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.971991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.998885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.024875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.427634] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1172.777559] LNet: 64903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.782534] LNetError: 64903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.791722] LNet: Removed LNI 192.168.204.11@tcp [ 1172.803537] LNet: Removed LNI 192.168.204.11@tcp1 [ 1173.300490] Key type .llcrypt unregistered [ 1173.304052] Key type ._llcrypt unregistered [ 1191.379262] Key type ._llcrypt registered [ 1191.380773] Key type .llcrypt registered [ 1192.546481] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:47:04 (1787381224) [ 1192.989059] LNet: 65704:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1192.989059] mask usage: [+|-] ... [ 1193.209042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1193.234227] alg: No test for adler32 (adler32-zlib) [ 1194.241530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.330736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1194.363026] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1195.561467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1195.606510] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1196.814735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1196.921788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1196.986363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1197.034808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1197.105238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1197.130519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1197.155909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1197.178479] LNetError: 66003:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1197.407377] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1197.416965] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1199.456171] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1199.625935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1203.551409] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1204.527262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1204.581635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1204.649608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1204.668467] LNet: 66110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.673309] LNetError: 66110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.683659] LNet: Removed LNI 192.168.204.11@tcp [ 1204.692334] LNet: Removed LNI 192.168.204.11@tcp1 [ 1204.718966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.794734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1204.829657] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1206.003913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1206.027910] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1206.948610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1206.983290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1207.023912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1207.064254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1207.092473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1207.116665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1207.150353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1207.189698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1207.216760] LNetError: 66383:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1207.840096] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1210.802481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1213.984071] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1213.992123] LNet: Skipped 1 previous similar message [ 1217.710515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1217.784610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.829756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.946509] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:47:36 (1787381256) [ 1225.135790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1225.177803] LNet: 67089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.182880] LNetError: 67089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.187105] LNetError: 67089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1225.191993] LNet: Removed LNI 192.168.204.11@tcp [ 1225.205551] LNet: Removed LNI 192.168.204.11@tcp1 [ 1225.230369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.283913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1225.306414] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1225.996575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1226.019165] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1226.829490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1226.882249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1226.912685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1226.956658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1226.980922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1227.021527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1227.059803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1227.143283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1227.295175] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.11@tcp [ 1228.319453] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1228.325107] LNet: Skipped 1 previous similar message [ 1243.591978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1243.770858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.828518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.877918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 500 [ 1244.063144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1250.957527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1251.030814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1251.061439] LNet: 67589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.067311] LNetError: 67589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.072518] LNetError: 67589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1251.079687] LNet: Removed LNI 192.168.204.11@tcp [ 1251.112039] LNet: Removed LNI 192.168.204.11@tcp1 [ 1251.154906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.210353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1251.252417] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1252.422527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1252.453763] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1253.428220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1253.499308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1253.552180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1253.588486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1253.629639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1253.670942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1253.715960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1253.773815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.11@tcp --health 0 [ 1254.239363] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1254.251681] LNet: Skipped 3 previous similar messages [ 1255.144637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.669967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1265.746337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1265.807506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1273.909638] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:48:25 (1787381305) [ 1274.779654] LNet: 68635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1274.813344] LNetError: 68635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.822490] LNetError: 68635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1275.872886] LNet: Removed LNI 192.168.204.11@tcp [ 1275.894800] LNet: Removed LNI 192.168.204.11@tcp1 [ 1276.912873] Key type .llcrypt unregistered [ 1276.920967] Key type ._llcrypt unregistered [ 1284.593758] Key type ._llcrypt registered [ 1284.597414] Key type .llcrypt registered [ 1284.644804] LNet: 68859:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1284.644804] mask usage: [+|-] ... [ 1284.929769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1284.954796] alg: No test for adler32 (adler32-zlib) [ 1286.019779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1286.067742] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1296.462609] Lustre: DEBUG MARKER: Initial discovery [ 1296.779034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1299.240208] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1299.407202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1299.490437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1299.545424] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1299.607568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1299.665706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -l 3 [ 1299.733872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.111@tcp -r 1 -m GET -e local_error [ 1299.789284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.111@tcp -r 1 -m PUT -l 6 [ 1300.450107] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1301.772626] Lustre: DEBUG MARKER: Force oleg411-client.virtnet to discover 192.168.204.111@tcp (in background) [ 1301.893895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.111@tcp [ 1303.636529] Lustre: DEBUG MARKER: Force oleg411-server to discover 192.168.204.11@tcp [ 1304.991176] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1306.750335] Lustre: DEBUG MARKER: Wait for 69395 [ 1308.344160] Lustre: DEBUG MARKER: Finished wait on 69395 [ 1308.643504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1308.707840] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1308.723839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1309.106619] LNet: 69848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.112096] LNetError: 69848:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.148174] LNet: Removed LNI 192.168.204.11@tcp [ 1309.604545] Key type .llcrypt unregistered [ 1309.617377] Key type ._llcrypt unregistered [ 1328.906352] Key type ._llcrypt registered [ 1328.912588] Key type .llcrypt registered [ 1330.543380] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:49:21 (1787381361) [ 1331.203435] Key type .llcrypt unregistered [ 1331.206666] Key type ._llcrypt unregistered [ 1337.505230] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1337.625510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1337.629984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1337.644355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1340.135418] Key type ._llcrypt registered [ 1340.141515] Key type .llcrypt registered [ 1340.180229] LNet: 70887:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1340.180229] mask usage: [+|-] ... [ 1340.299264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.318221] alg: No test for adler32 (adler32-zlib) [ 1341.249500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.303175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1341.329131] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1342.092843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1342.116184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1351.540528] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:49:43 (1787381383) [ 1351.726052] LNet: 71913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.734908] LNetError: 71913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1351.746949] LNet: Removed LNI 10.1.2.3@tcp [ 1351.798110] LNet: Removed LNI 192.168.204.11@tcp [ 1352.695726] Key type .llcrypt unregistered [ 1352.702423] Key type ._llcrypt unregistered [ 1360.855393] Key type ._llcrypt registered [ 1360.858834] Key type .llcrypt registered [ 1360.955383] LNet: 72170:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1360.955383] mask usage: [+|-] ... [ 1361.243764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1361.270929] alg: No test for adler32 (adler32-zlib) [ 1362.290522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.379368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.424315] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1363.915877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1363.946594] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1365.521916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.11@tcp [ 1366.625725] LNetError: Unexpected error -22 connecting to 192.168.204.11@tcp at host 192.168.204.11:988 [ 1377.198864] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:50:08 (1787381408) [ 1377.651322] LNet: 73054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.662857] LNetError: 73054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1377.674113] LNet: Removed LNI 10.1.2.3@tcp [ 1377.722885] LNet: Removed LNI 192.168.204.11@tcp [ 1378.959488] Key type .llcrypt unregistered [ 1378.963750] Key type ._llcrypt unregistered [ 1388.497415] Key type ._llcrypt registered [ 1388.509658] Key type .llcrypt registered [ 1388.569792] LNet: 73285:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1388.569792] mask usage: [+|-] ... [ 1388.868478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1388.879844] alg: No test for adler32 (adler32-zlib) [ 1389.831330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.882683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.901898] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1390.714938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1390.766213] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 1392.034338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.11@tcp1 --nid 192.168.204.11@tcp2 [ 1396.440307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp2 [ 1396.465121] LNetError: 73890:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.11@tcp1 to 192.168.204.11@tcp2 - Route table may be misconfigured [ 1396.508622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp2 192.168.204.11@tcp1 [ 1404.772252] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:50:36 (1787381436) [ 1404.904910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.938605] LNet: 74456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.952278] LNetError: 74456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.958520] LNet: Removed LNI 192.168.204.11@tcp1 [ 1405.990034] LNet: Removed LNI 192.168.204.11@tcp2 [ 1406.090972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.249888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.352917] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1407.591748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.619882] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1408.917649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1409.000834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.11@tcp [ 1409.063465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp1 192.168.204.11@tcp1 [ 1409.138667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp -e network_timeout [ 1409.207069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -e network_timeout [ 1409.251860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -e network_timeout [ 1409.308875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -e network_timeout [ 1409.377291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1409.418451] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1409.459392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1409.569809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1409.743154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1410.275536] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1417.729495] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:50:49 (1787381449) [ 1418.076339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.124778] LNet: 75326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1418.137706] LNetError: 75326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1418.149111] LNetError: 75326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1418.159375] LNet: Removed LNI 192.168.204.11@tcp [ 1419.244910] LNet: Removed LNI 192.168.204.11@tcp1 [ 1419.317251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.447846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1419.649965] LNet: 75363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.972269] Key type .llcrypt unregistered [ 1419.976822] Key type ._llcrypt unregistered [ 1429.813362] Key type ._llcrypt registered [ 1429.815988] Key type .llcrypt registered [ 1430.815869] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:51:02 (1787381462) [ 1431.314364] LNet: 76135:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1431.314364] mask usage: [+|-] ... [ 1431.450895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.474877] alg: No test for adler32 (adler32-zlib) [ 1432.394984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.873220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1435.508615] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:51:07 (1787381467) [ 1435.703258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.737965] LNet: 76458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.788498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.874839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.900130] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1436.686878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.716925] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1437.548417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1437.591905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp1 [ 1437.634152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp1 [ 1443.578755] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:51:15 (1787381475) [ 1445.561195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg411-server [ 1447.259757] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:51:19 (1787381479) [ 1449.270934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg411-server [ 1450.772088] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:51:22 (1787381482) [ 1452.418818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg411-server [ 1454.131569] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:51:26 (1787381486) [ 1455.840314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg411-server [ 1457.679068] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:51:29 (1787381489) [ 1459.581489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg411-server [ 1461.306947] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:51:33 (1787381493) [ 1463.011884] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg411-server [ 1464.746597] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:51:36 (1787381496) [ 1466.722825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg411-server [ 1468.587536] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:51:40 (1787381500) [ 1470.435902] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg411-server [ 1472.149191] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:51:44 (1787381504) [ 1473.935431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg411-server [ 1475.764398] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:51:47 (1787381507) [ 1475.894941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.910118] LNet: 80337:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.918219] LNetError: 80337:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.923522] LNet: Removed LNI 192.168.204.11@tcp [ 1476.968880] LNet: Removed LNI 192.168.204.11@tcp1 [ 1477.015777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.127816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.146614] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1477.955558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1478.058593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1478.210619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.247786] LNet: 80486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.267490] LNetError: 80486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.275194] LNetError: 80486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1478.303390] LNet: Removed LNI 192.168.204.11@tcp [ 1478.323604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.392901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1478.433091] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1479.275491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1479.500686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.529963] LNetError: 80632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.556333] LNet: Removed LNI 192.168.204.11@tcp [ 1479.593142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.640408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.661964] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1480.337152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1480.456102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1480.666381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.688246] LNet: 80779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.692307] LNet: 80779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1481.768013] LNet: Removed LNI 192.168.204.11@tcp [ 1481.813675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.879880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1481.933616] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1482.772314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1483.074307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1483.097623] LNetError: 80924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.103628] LNetError: 80924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1484.134406] LNet: Removed LNI 192.168.204.11@tcp [ 1484.171561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.261622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.297452] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1484.972248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1485.075251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1485.386050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.414656] LNet: 81072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.421990] LNet: 81072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1486.438732] LNet: Removed LNI 192.168.204.11@tcp [ 1486.506892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.626618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1486.670731] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1487.578901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1487.833868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.869861] LNetError: 81217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.886690] LNetError: 81217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1487.907718] LNet: Removed LNI 192.168.204.11@tcp [ 1487.958104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.025323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.067579] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1488.936258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1489.057928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1489.394525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.479989] LNet: Removed LNI 192.168.204.11@tcp [ 1490.535428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.664638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1490.708801] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1491.685945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1492.079874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.173136] LNet: Removed LNI 192.168.204.11@tcp [ 1493.232789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.369768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.430162] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1494.461560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1494.582414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1494.945751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.987514] LNet: 81658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.992948] LNet: 81658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1496.041656] LNet: Removed LNI 192.168.204.11@tcp [ 1496.102121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.227474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1496.252741] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1497.465475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1497.906635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.954402] LNetError: 81803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.970085] LNetError: 81803:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1498.004431] LNet: Removed LNI 192.168.204.11@tcp [ 1498.029919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.110131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1498.173630] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1498.989411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1499.095469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1499.485354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.584683] LNet: Removed LNI 192.168.204.11@tcp [ 1500.609651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.678263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1500.705363] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1501.424551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1501.728644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.827319] LNet: Removed LNI 192.168.204.11@tcp [ 1502.862975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.944047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.987754] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1503.776734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1503.844903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1503.874198] LNet: Removed LNI 192.168.204.11@tcp [ 1503.890222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.951397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.003821] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1504.833532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1512.038581] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:52:23 (1787381543) [ 1512.266040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.279778] LNet: 82933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.284494] LNet: 82933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1512.301066] LNet: Removed LNI 192.168.204.11@tcp [ 1512.327378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.408994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1512.443850] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1513.828452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.867566] LNet: Removed LNI 192.168.204.11@tcp231 [ 1513.891570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.948866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1514.001754] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1514.014591] LNet: 83079:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1514.433863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1514.464618] LNet: Removed LNI 192.168.204.11@tcp231 [ 1514.481908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1514.503125] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1515.823305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.850241] LNetError: 83224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.857540] LNetError: 83224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1515.869882] LNet: Removed LNI 192.168.204.11@tcp231 [ 1515.916335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.996700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1516.027911] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1516.032886] LNet: 83232:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1516.441893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1516.486975] LNet: Removed LNI 192.168.204.11@tcp231 [ 1516.502693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1516.534502] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/0] [ 1517.617209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.657961] LNet: Removed LNI 192.168.204.11@tcp231 [ 1517.709728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.788946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1517.852094] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1517.858638] LNet: 83383:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1518.339733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1518.389024] LNet: Removed LNI 192.168.204.11@tcp231 [ 1518.410499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1518.444926] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/60] [ 1519.618221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.657770] LNet: Removed LNI 192.168.204.11@tcp231 [ 1519.691578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.775996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1519.814863] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1520.306950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1520.346684] LNet: Removed LNI 192.168.204.11@tcp231 [ 1520.359748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1520.379818] LNet: Added LNI 192.168.204.11@tcp231 [8/256/0/180] [ 1527.114387] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:52:38 (1787381558) [ 1527.175388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.213948] LNet: Removed LNI 192.168.204.11@tcp231 [ 1527.260640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.353142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1527.377524] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1528.252191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1528.317600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1528.353160] LNet: Removed LNI 192.168.204.11@tcp232 [ 1528.382063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1528.454428] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1529.214852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1529.265980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1529.294990] LNet: Removed LNI 192.168.204.11@tcp232 [ 1529.321466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1529.352650] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1530.033860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1530.076785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1530.107640] LNet: Removed LNI 192.168.204.11@tcp232 [ 1530.124365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1530.162347] LNet: Added LNI 192.168.204.11@tcp232 [8/256/0/180] [ 1531.126964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1531.183513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1531.230805] LNet: Removed LNI 192.168.204.11@tcp232 [ 1531.250290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1538.175290] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:52:49 (1787381569) [ 1539.445690] Key type .llcrypt unregistered [ 1539.448262] Key type ._llcrypt unregistered [ 1545.870729] Key type ._llcrypt registered [ 1545.876507] Key type .llcrypt registered [ 1545.931582] LNet: 85602:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1545.931582] mask usage: [+|-] ... [ 1546.146702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1546.165264] alg: No test for adler32 (adler32-zlib) [ 1547.124568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1547.167343] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1555.841642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1565.211776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.257767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1565.276664] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1566.794259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1566.860892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1566.895937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1566.944554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1567.001371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_interrupt -r 2 [ 1567.051484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_interrupt -r 2 [ 1567.087805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_interrupt -r 2 [ 1567.151454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_interrupt -r 2 [ 1567.199810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1567.251721] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1567.521087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1567.559786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1567.586046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1567.613825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1567.645895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1567.683744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1567.720858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1567.751974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1567.782963] LNet: 86035:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.812854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1567.855293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.894590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1567.927336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.964790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1567.997712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1568.543182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1568.588159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1568.656080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1568.710387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1568.739324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_dropped -r 2 [ 1568.797694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_dropped -r 2 [ 1568.834712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_dropped -r 2 [ 1568.874236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_dropped -r 2 [ 1568.922668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1568.948781] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1568.954859] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1569.210161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1569.253496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1569.297854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1569.337262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1569.379336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1569.433263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1569.468516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1569.512571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1569.547946] LNet: 86155:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1569.563740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1569.605177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1569.635543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1569.670634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1569.718459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1569.760194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1570.381628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1570.443087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1570.494391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1570.590629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1570.672989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_aborted -r 2 [ 1570.721743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_aborted -r 2 [ 1570.762975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_aborted -r 2 [ 1570.815901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_aborted -r 2 [ 1570.866442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1570.898254] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1571.158719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1571.195696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1571.235452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1571.275733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1571.326846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1571.384836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1571.431988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1571.481244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1571.508545] LNet: 86276:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1571.536744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1571.589692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1571.651751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1571.723241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1571.771488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1571.801708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1572.436359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1572.495977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1572.537815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1572.575792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1572.629574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_no_route -r 2 [ 1572.680334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_no_route -r 2 [ 1572.724992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_no_route -r 2 [ 1572.773976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_no_route -r 2 [ 1572.821739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1573.202851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1573.260854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1573.297968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1573.349706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1573.405853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1573.444971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1573.497188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1573.599865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1573.632625] LNet: 86396:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1573.657285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1573.709934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1573.752812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1573.795174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1573.839131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1573.872390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1574.511647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1574.559561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1574.615841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1574.677151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1574.759568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e local_timeout -r 2 [ 1574.823026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e local_timeout -r 2 [ 1574.868949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e local_timeout -r 2 [ 1574.916952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e local_timeout -r 2 [ 1574.952123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1574.973514] LNet: There was an unexpected network error while writing to 192.168.204.111: rc = -22 [ 1574.979974] LNet: Skipped 1 previous similar message [ 1575.265124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1575.302648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1575.336891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1575.392568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1575.434720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1575.494084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1575.549669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1575.591542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1575.628676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1575.671796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1575.729288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1575.774778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1575.822965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1575.871563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1576.483473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1576.530823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1576.564482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1576.602487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1576.665326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp -m GET -e remote_dropped -r 2 [ 1576.713529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.111@tcp1 -m GET -e remote_dropped -r 2 [ 1576.784549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp -m GET -e remote_dropped -r 2 [ 1576.850606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.11@tcp1 -d 192.168.204.11@tcp1 -m GET -e remote_dropped -r 2 [ 1576.912493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.11@tcp 192.168.204.111@tcp [ 1576.931079] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 1577.150612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1577.178991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1577.221742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.265293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.324614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.352843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.375731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.406407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.450459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.502618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.548335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.597898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1577.637321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.683955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1583.195539] LNet: 86710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.204171] LNetError: 86710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.216630] LNet: Removed LNI 192.168.204.11@tcp [ 1583.226365] LNet: Removed LNI 192.168.204.11@tcp1 [ 1583.723617] Key type .llcrypt unregistered [ 1583.726802] Key type ._llcrypt unregistered [ 1601.888308] Key type ._llcrypt registered [ 1601.889973] Key type .llcrypt registered [ 1603.205128] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:53:55 (1787381635) [ 1603.382539] LNet: 87516:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1603.382539] mask usage: [+|-] ... [ 1603.576064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.592988] alg: No test for adler32 (adler32-zlib) [ 1604.529229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.618296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.639227] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1605.455466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1605.489607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.11@tcp [ 1605.513288] LNetError: 87689:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1605.587873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.11@tcp [ 1605.599517] LNet: 1 local NIs in recovery (showing 1): 192.168.204.11@tcp [ 1605.721398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1612.817164] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:54:04 (1787381644) [ 1612.932354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.950378] LNet: 88271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.960053] LNetError: 88271:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.984921] LNet: Removed LNI 192.168.204.11@tcp [ 1613.018690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.111680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1613.161831] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1614.023256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1614.047507] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1614.997647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.11@tcp [ 1615.034887] LNetError: 88544:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.204.11@tcp to recovery [ 1615.136776] LNet: There was an unexpected network error while writing to 192.168.204.11: rc = -22 [ 1616.301617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.11@tcp [ 1623.848828] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:54:15 (1787381655) [ 1626.018690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg411-server [ 1628.073156] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:54:19 (1787381659) [ 1630.645683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg411-server [ 1632.281619] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:54:24 (1787381664) [ 1634.655954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg411-server [ 1636.529547] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:54:28 (1787381668) [ 1636.734127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.786571] LNet: 90166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.797230] LNetError: 90166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.806255] LNet: Removed LNI 192.168.204.11@tcp [ 1636.822279] LNet: Removed LNI 192.168.204.11@tcp1 [ 1636.862686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.933844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.961542] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1637.839468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1637.962724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1638.054461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1638.193362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1638.245470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1638.303605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1645.676637] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:54:37 (1787381677) [ 1645.894360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.935282] LNet: 90893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.939969] LNetError: 90893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.945822] LNetError: 90893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1645.965448] LNet: Removed LNI 192.168.204.11@tcp [ 1645.983216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.037578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.070197] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1654.730266] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:54:46 (1787381686) [ 1655.065267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.107386] LNet: 91596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.113512] LNetError: 91596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.146226] LNet: Removed LNI 192.168.204.11@tcp [ 1655.185185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.271558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.304482] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1663.904208] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:54:55 (1787381695) [ 1665.869484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1667.672335] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:54:59 (1787381699) [ 1668.538068] LNet: 92632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.550135] LNetError: 92632:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.578704] LNet: Removed LNI 192.168.204.11@tcp [ 1669.270343] Key type .llcrypt unregistered [ 1669.275832] Key type ._llcrypt unregistered [ 1677.728881] Key type ._llcrypt registered [ 1677.731259] Key type .llcrypt registered [ 1677.773723] LNet: 92861:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1677.773723] mask usage: [+|-] ... [ 1677.984699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1678.008921] alg: No test for adler32 (adler32-zlib) [ 1678.992113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1679.048353] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1688.308455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1694.264423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.111@tcp [ 1694.711809] LNet: 93051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.726922] LNetError: 93051:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.777058] LNet: Removed LNI 192.168.204.11@tcp [ 1695.306776] Key type .llcrypt unregistered [ 1695.311410] Key type ._llcrypt unregistered [ 1708.484789] Key type ._llcrypt registered [ 1708.486919] Key type .llcrypt registered [ 1710.187259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1712.003784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1714.002518] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:55:45 (1787381745) [ 1716.116537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1718.059845] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:55:49 (1787381749) [ 1721.001801] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg411-server [ 1723.350085] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:55:54 (1787381754) [ 1726.502404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg411-server [ 1728.645752] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:56:00 (1787381760) [ 1729.953401] LNet: 95104:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1729.953401] mask usage: [+|-] ... [ 1730.796862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1730.907633] alg: No test for adler32 (adler32-zlib) [ 1732.184950] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 1732.613852] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1734.399224] Key type lgssc registered [ 1736.256425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1760.016605] Key type lgssc unregistered [ 1760.322527] LNet: 96228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.331584] LNetError: 96228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.347444] LNet: Removed LNI 192.168.204.11@tcp [ 1761.167078] Key type .llcrypt unregistered [ 1761.175133] Key type ._llcrypt unregistered [ 1776.911489] Key type ._llcrypt registered [ 1776.913763] Key type .llcrypt registered [ 1778.227409] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:56:50 (1787381810) [ 1778.487637] LNet: 97026:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1778.487637] mask usage: [+|-] ... [ 1778.653700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.665817] alg: No test for adler32 (adler32-zlib) [ 1779.587809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.650079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1779.713684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1779.794869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1779.842774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1786.354558] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:56:58 (1787381818) [ 1786.615613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.667279] LNet: 97644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.692871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.770521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.816388] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1788.074851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.098788] LNet: 97789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.109537] LNetError: 97789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.143597] LNet: Removed LNI 192.168.204.11@tcp [ 1788.176590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.280507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1788.310254] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1788.756182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.775747] LNetError: 97817:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.787826] LNet: Removed LNI 192.168.204.11@tcp [ 1788.806927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.844512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1788.869744] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1797.234217] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:57:08 (1787381828) [ 1797.610416] LNet: 98408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.616330] LNet: 98408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1797.626485] LNetError: 98408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.640421] LNet: Removed LNI 192.168.204.11@tcp [ 1798.718664] Key type .llcrypt unregistered [ 1798.722880] Key type ._llcrypt unregistered [ 1808.808357] Key type ._llcrypt registered [ 1808.810447] Key type .llcrypt registered [ 1808.836636] LNet: 98651:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1808.836636] mask usage: [+|-] ... [ 1808.962936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.977073] alg: No test for adler32 (adler32-zlib) [ 1809.931587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.021036] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1810.023195] LNet: *** cfs_fail_loc=e003, val=100*** [ 1810.686196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1810.713535] LNet: *** cfs_fail_loc=e003, val=100*** [ 1810.716738] LNet: Skipped 2 previous similar messages [ 1810.725909] Ping reply received with mapping: [ 1810.728155] found metadata reply NI[192.168.204.11@tcp] [ 1810.734927] 0b cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1810.740329] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1810.745259] 5a 5a 5a 5a 5a 5a 5a 5a [ 1810.889433] LNet: 98817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.894070] LNetError: 98817:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1811.942843] LNet: Removed LNI 192.168.204.11@tcp [ 1812.561474] Key type .llcrypt unregistered [ 1812.564593] Key type ._llcrypt unregistered [ 1825.883805] Key type ._llcrypt registered [ 1825.886562] Key type .llcrypt registered [ 1827.247368] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:57:38 (1787381858) [ 1828.109551] Key type .llcrypt unregistered [ 1828.116160] Key type ._llcrypt unregistered [ 1835.557707] Key type ._llcrypt registered [ 1835.559962] Key type .llcrypt registered [ 1835.598447] LNet: 99815:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1835.598447] mask usage: [+|-] ... [ 1835.821768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1835.839815] alg: No test for adler32 (adler32-zlib) [ 1837.010865] LNetError: 99865:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1837.459338] Key type .llcrypt unregistered [ 1837.461654] Key type ._llcrypt unregistered [ 1842.696207] Key type ._llcrypt registered [ 1842.709908] Key type .llcrypt registered [ 1844.206970] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:57:55 (1787381875) [ 1844.903458] Key type .llcrypt unregistered [ 1844.906816] Key type ._llcrypt unregistered [ 1853.082622] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1853.089223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1855.702253] Key type ._llcrypt registered [ 1855.710169] Key type .llcrypt registered [ 1855.770918] LNet: 100733:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1855.770918] mask usage: [+|-] ... [ 1855.933742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1855.948595] alg: No test for adler32 (adler32-zlib) [ 1856.912568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.010312] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1866.892174] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:58:18 (1787381898) [ 1866.958802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.978457] LNet: 101639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.996817] LNetError: 101639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1867.014607] LNet: Removed LNI 10.1.2.3@tcp [ 1867.049543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.750186] Key type .llcrypt unregistered [ 1867.755386] Key type ._llcrypt unregistered [ 1874.157853] Key type ._llcrypt registered [ 1874.160467] Key type .llcrypt registered [ 1874.238811] LNet: 101878:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1874.238811] mask usage: [+|-] ... [ 1874.506196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1874.531505] alg: No test for adler32 (adler32-zlib) [ 1875.545027] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1875.639150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1875.850319] LNet: 101935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.856212] LNetError: 101935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.889992] LNet: Removed LNI 192.168.204.11@tcp [ 1876.469972] Key type .llcrypt unregistered [ 1876.472521] Key type ._llcrypt unregistered [ 1887.508650] Key type ._llcrypt registered [ 1887.511583] Key type .llcrypt registered [ 1888.882549] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:58:40 (1787381920) [ 1889.583409] Key type .llcrypt unregistered [ 1889.588770] Key type ._llcrypt unregistered [ 1896.388390] Key type ._llcrypt registered [ 1896.390049] Key type .llcrypt registered [ 1896.440234] LNet: 102920:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1896.440234] mask usage: [+|-] ... [ 1896.673404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1896.696905] alg: No test for adler32 (adler32-zlib) [ 1903.069555] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:58:54 (1787381934) [ 1903.204509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.270251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.330748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1903.377076] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1904.167628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.198465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1904.227970] LNet: Removed LNI 192.168.204.11@tcp [ 1904.420383] LNet: 103679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.771581] Key type .llcrypt unregistered [ 1904.778069] Key type ._llcrypt unregistered [ 1914.721692] Key type ._llcrypt registered [ 1914.725744] Key type .llcrypt registered [ 1916.073616] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:59:07 (1787381947) [ 1916.328103] LNet: 104446:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1916.328103] mask usage: [+|-] ... [ 1916.464793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.479926] alg: No test for adler32 (adler32-zlib) [ 1917.416365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.481444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.505994] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1918.129089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 1918.171419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.11@tcp [ 1918.199791] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1918.205750] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.209336] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 0 [ 1918.215523] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1918.218702] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1918.221152] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.226163] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 0 [ 1918.231935] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1918.235460] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.239282] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.242585] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.248245] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1918.252247] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.256734] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.261575] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.267464] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1924.572383] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:59:16 (1787381956) [ 1925.030534] LNet: 105221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.036932] LNetError: 105221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.059093] LNet: Removed LNI 192.168.204.11@tcp [ 1925.767330] Key type .llcrypt unregistered [ 1925.770828] Key type ._llcrypt unregistered [ 1932.660251] Key type ._llcrypt registered [ 1932.663375] Key type .llcrypt registered [ 1932.704536] LNet: 105444:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1932.704536] mask usage: [+|-] ... [ 1932.871194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.890431] alg: No test for adler32 (adler32-zlib) [ 1933.837416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1933.874552] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1941.568199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 1951.123050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.169332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1951.190230] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 1952.306036] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1952.495529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1952.510814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1952.523761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1954.920188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1954.942805] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1966.087313] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:59:57 (1787381997) [ 1966.253884] LNet: 106722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.270058] LNetError: 106722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.281558] LNet: Removed LNI 192.168.204.11@tcp [ 1967.327469] LNet: Removed LNI 192.168.204.11@tcp1 [ 1967.336267] LNetError: 106722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1967.361263] LNetError: 106722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1967.385103] LNet: Removed LNI 10.1.2.3@tcp99 [ 1968.119724] Key type .llcrypt unregistered [ 1968.126216] Key type ._llcrypt unregistered [ 1975.718906] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1975.727775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1978.243333] Key type ._llcrypt registered [ 1978.246330] Key type .llcrypt registered [ 1978.302324] LNet: 106992:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1978.302324] mask usage: [+|-] ... [ 1978.540548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1978.560754] alg: No test for adler32 (adler32-zlib) [ 1979.575364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.660223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1979.706301] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 1980.681730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1980.717067] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1984.083990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1984.155720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1984.233685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1984.270685] LNetError: 107471:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1984.321675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1984.397089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 1984.834168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1984.989478] LNet: 107491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.002193] LNetError: 107491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1985.011633] LNet: Removed LNI 10.1.2.3@tcp [ 1986.087618] LNet: Removed LNI 192.168.204.11@tcp [ 1987.247581] Key type .llcrypt unregistered [ 1987.249132] Key type ._llcrypt unregistered [ 2002.001376] Key type ._llcrypt registered [ 2002.005315] Key type .llcrypt registered [ 2003.268080] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:00:35 (1787382035) [ 2003.681953] LNet: 108291:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2003.681953] mask usage: [+|-] ... [ 2003.898856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.914475] alg: No test for adler32 (adler32-zlib) [ 2004.821187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.869103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.899863] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2005.554480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.11@tcp [ 2005.616572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg411-client@tcp [ 2012.567098] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:00:44 (1787382044) [ 2012.733603] LNet: 109018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.751401] LNetError: 109018:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.784362] LNet: Removed LNI 192.168.204.11@tcp [ 2013.623690] Key type .llcrypt unregistered [ 2013.628528] Key type ._llcrypt unregistered [ 2021.015874] Key type ._llcrypt registered [ 2021.019691] Key type .llcrypt registered [ 2021.307660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.321734] alg: No test for adler32 (adler32-zlib) [ 2022.318684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2022.354199] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2022.390221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2030.908717] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:01:02 (1787382062) [ 2032.719982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2034.626649] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:01:06 (1787382066) [ 2036.333828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2038.315028] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:01:09 (1787382069) [ 2038.425588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.458583] LNet: 110453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.464293] LNetError: 110453:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.485147] LNet: Removed LNI 192.168.204.11@tcp [ 2038.500071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.552343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.582563] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2039.473286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2039.544077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2046.413410] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:01:18 (1787382078) [ 2046.492046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.522475] LNet: 111161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.550842] LNetError: 111161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.591686] LNet: Removed LNI 192.168.204.11@tcp [ 2046.660397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.735823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2046.779443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2047.473626] Key type .llcrypt unregistered [ 2047.477515] Key type ._llcrypt unregistered [ 2059.124581] Key type ._llcrypt registered [ 2059.126256] Key type .llcrypt registered [ 2060.561643] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:01:32 (1787382092) [ 2060.903568] LNet: 111976:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2060.903568] mask usage: [+|-] ... [ 2061.042415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.060682] alg: No test for adler32 (adler32-zlib) [ 2062.081636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.174981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2062.206940] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2063.051502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2063.132882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2069.723937] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:01:41 (1787382101) [ 2069.971637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.997823] LNet: 112727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.005643] LNetError: 112727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.036981] LNet: Removed LNI 192.168.204.11@tcp [ 2070.089617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2070.169780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2070.229707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2076.859196] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:01:48 (1787382108) [ 2077.043635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.061775] LNet: 113307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.077652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.132146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2077.598825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2077.671850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2077.779619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2078.283733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2078.354793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2078.437423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.475164] LNet: 113340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2078.509374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.612786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2079.074887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2079.107032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2079.156463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2079.762358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2079.796515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2086.582397] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:01:58 (1787382118) [ 2086.707910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.730795] LNet: 113937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.785430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.880460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2086.929487] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2087.922716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2087.951642] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2088.943372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2089.040570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2089.214399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2089.231808] LNetError: 114221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.272156] LNet: Removed LNI 192.168.204.11@tcp [ 2090.278714] LNetError: 114221:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.304847] LNet: Removed LNI 192.168.204.11@tcp2 [ 2090.346164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.422877] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2090.441500] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2090.449962] LNet: 114227:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2091.147576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2091.400576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.424314] LNet: 114264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.439350] LNet: 114264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2092.511166] LNet: Removed LNI 192.168.204.11@tcp [ 2092.516347] LNetError: 114264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.543237] LNetError: 114264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2092.573742] LNet: Removed LNI 192.168.204.11@tcp2 [ 2092.616456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.697915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2092.723876] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2093.633866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2093.678652] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2094.632975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2094.708978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2094.810422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2094.997572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.096031] LNet: Removed LNI 192.168.204.11@tcp [ 2097.127521] LNet: Removed LNI 192.168.204.11@tcp2 [ 2097.170643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.214505] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2097.219111] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2097.222797] LNet: 114555:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2097.836656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2098.143161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.172386] LNetError: 114592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2098.177183] LNetError: 114592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2098.182527] LNet: Removed LNI 192.168.204.11@tcp [ 2099.238337] LNet: Removed LNI 192.168.204.11@tcp2 [ 2099.262169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.361530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2099.406858] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2100.161421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2100.186942] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2100.974801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2101.032714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2101.246634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.262794] LNet: 114872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.271190] LNet: 114872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2102.303910] LNet: Removed LNI 192.168.204.11@tcp [ 2102.317711] LNet: Removed LNI 192.168.204.11@tcp2 [ 2102.370376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.489115] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2102.502416] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2102.510366] LNet: 114878:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.347116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2103.609463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.671231] LNet: Removed LNI 192.168.204.11@tcp [ 2104.688059] LNet: Removed LNI 192.168.204.11@tcp2 [ 2104.735902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.834317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2104.874490] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2105.776895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2105.795203] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2106.810595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2106.844717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2106.902404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2107.159605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.181202] LNetError: 115200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.187527] LNetError: 115200:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2107.193934] LNet: Removed LNI 192.168.204.11@tcp [ 2108.261696] LNet: Removed LNI 192.168.204.11@tcp2 [ 2108.281570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.345524] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2108.357478] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2108.365495] LNet: 115206:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2109.054971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2109.340513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.374544] LNet: Removed LNI 192.168.204.11@tcp [ 2110.444672] LNet: Removed LNI 192.168.204.11@tcp2 [ 2110.518131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.591671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2110.630325] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2111.544154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2111.574324] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2112.619662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2112.680615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2112.727197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2113.104774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.145461] LNet: Removed LNI 192.168.204.11@tcp [ 2114.161069] LNet: Removed LNI 192.168.204.11@tcp2 [ 2114.194203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.278573] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2114.286443] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2114.293316] LNet: 115547:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2114.846856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2115.210648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.256685] LNet: Removed LNI 192.168.204.11@tcp [ 2116.272233] LNet: Removed LNI 192.168.204.11@tcp2 [ 2116.337335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.404513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.429538] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2117.243134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2117.279673] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2118.278694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2118.308088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2118.335115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2118.394513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2118.782232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.805417] LNet: 115900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.819322] LNet: 115900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2119.839752] LNet: Removed LNI 192.168.204.11@tcp [ 2120.870330] LNet: Removed LNI 192.168.204.11@tcp2 [ 2120.922949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.040965] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2121.055065] LNet: Added LNI 192.168.204.11@tcp2 [8/256/0/180] [ 2121.726839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2129.614696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2131.641846] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:02:43 (1787382163) [ 2131.751935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.802040] LNetError: 116665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.821875] LNetError: 116665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2131.838897] LNet: Removed LNI 192.168.204.11@tcp [ 2132.913120] LNet: Removed LNI 192.168.204.11@tcp2 [ 2132.942388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.221704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.243610] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2133.973953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2133.991660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2135.030074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.060263] LNet: Removed LNI 10.1.2.3@tcp [ 2135.068077] LNet: Removed LNI 192.168.204.11@tcp [ 2135.085480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.158339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2135.179515] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2135.185681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2135.757857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2135.795348] LNet: Removed LNI 192.168.204.11@tcp [ 2135.812716] LNet: Removed LNI 10.1.2.3@tcp [ 2142.092223] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:02:53 (1787382173) [ 2142.294527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.350582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.462605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2142.595311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2142.605950] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2142.615849] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2144.932401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.971253] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2146.028612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2146.043932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2148.308318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.11@tcp [ 2148.370798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2148.499692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.11@tcp -m PUT -e local_timeout -r 1 [ 2148.591579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2148.668920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2148.679912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2148.924293] LNetError: 117613:0:(lib-move.c:4714:lnet_parse()) 192.168.204.11@tcp, src 192.168.204.11@tcp: Dropping PUT (error -108 looking up sender) [ 2149.987415] LNet: Removed LNI 10.1.2.3@tcp [ 2151.015646] LNet: Removed LNI 192.168.204.11@tcp [ 2151.484417] Key type .llcrypt unregistered [ 2151.485903] Key type ._llcrypt unregistered [ 2164.528809] Key type ._llcrypt registered [ 2164.539485] Key type .llcrypt registered [ 2165.859534] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:03:17 (1787382197) [ 2166.055787] LNet: 118841:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2166.055787] mask usage: [+|-] ... [ 2166.325657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.361060] alg: No test for adler32 (adler32-zlib) [ 2167.366587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.829204] LNet: 118925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.689895] Key type .llcrypt unregistered [ 2168.695340] Key type ._llcrypt unregistered [ 2175.710828] Key type ._llcrypt registered [ 2175.717412] Key type .llcrypt registered [ 2175.756730] LNet: 119154:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2175.756730] mask usage: [+|-] ... [ 2175.934406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2175.953946] alg: No test for adler32 (adler32-zlib) [ 2176.935367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2176.968698] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2185.407482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2185.610647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2186.389261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2186.704668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2186.750323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2186.786655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2186.820734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2186.850369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2186.874345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2186.899798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2186.927173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2186.940217] LNet: 119376:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2186.951584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2186.985190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2187.017217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2187.049937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2187.095909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2187.130587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2187.877465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2188.130222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2188.167700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2188.200264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2188.235629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2188.271933] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.111@tcp [ 2188.276970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2188.311957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2188.349920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2188.395029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2188.443185] LNet: 119493:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2188.462878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2188.533858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2188.603310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2188.689538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2188.732312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2188.774450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2189.571884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2189.928827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2189.978436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2190.019165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2190.070471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2190.117528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2190.164933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2190.212923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2190.264504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2190.316845] LNet: 119610:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2190.352845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2190.419572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2190.464463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2190.527897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2190.599966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2190.682957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2191.498117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2191.855112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2191.945547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2192.008290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2192.077234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2192.158630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2192.197971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2192.233694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2192.269842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2192.312671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2192.350130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2192.386679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2192.422927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2192.486992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2192.540250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2193.044593] LNet: 119774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.056117] LNetError: 119774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.093868] LNet: Removed LNI 192.168.204.11@tcp [ 2193.707436] Key type .llcrypt unregistered [ 2193.711023] Key type ._llcrypt unregistered [ 2214.922780] Key type ._llcrypt registered [ 2214.925442] Key type .llcrypt registered [ 2216.267304] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:04:07 (1787382247) [ 2216.621855] LNet: 120581:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2216.621855] mask usage: [+|-] ... [ 2216.866946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2216.891413] alg: No test for adler32 (adler32-zlib) [ 2217.833957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.241573] LNet: 120663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.901456] Key type .llcrypt unregistered [ 2218.903571] Key type ._llcrypt unregistered [ 2225.978274] Key type ._llcrypt registered [ 2225.986435] Key type .llcrypt registered [ 2226.026197] LNet: 120892:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2226.026197] mask usage: [+|-] ... [ 2226.263655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2226.292279] alg: No test for adler32 (adler32-zlib) [ 2227.247678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2227.272667] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2235.517470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.111@tcp [ 2246.026888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.075984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2246.119845] LNet: Added LNI 192.168.204.11@tcp1 [8/256/0/180] [ 2247.186727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2247.761201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.111@tcp [ 2248.048773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2248.101000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2248.146781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2248.179370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2248.213712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2248.243569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2248.268546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2248.314548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2248.339470] LNet: 121317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.374336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2248.420636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2248.476231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2248.507967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.547826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.588773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2249.235157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2249.531808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.564804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2249.604404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.649753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.679436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.709885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.748649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.779942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.806830] LNet: 121434:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.825204] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.111@tcp, 192.168.204.111@tcp1 [ 2249.838972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2249.887462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.925436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.954408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.986667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2250.035548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2250.749600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.111@tcp [ 2251.016284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.059815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2251.096174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.144651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.183255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.225173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.276879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2251.301406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.319379] LNet: 121551:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.337880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.370441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.433548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.486648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.537098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.601857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.276741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2252.506960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.532567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2252.554445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.577459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.601283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.622328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.648685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.677035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.706919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2252.730077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2252.751496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2252.783086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2252.815553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2252.841082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2257.961368] LNet: 121749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.971422] LNetError: 121749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.985822] LNet: Removed LNI 192.168.204.11@tcp [ 2257.995496] LNet: Removed LNI 192.168.204.11@tcp1 [ 2258.395600] Key type .llcrypt unregistered [ 2258.400908] Key type ._llcrypt unregistered [ 2275.850871] Key type ._llcrypt registered [ 2275.859314] Key type .llcrypt registered [ 2277.424709] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:05:08 (1787382308) [ 2286.011586] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:05:17 (1787382317) [ 2292.981837] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:05:24 (1787382324) [ 2300.464463] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:05:32 (1787382332) [ 2307.202517] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:05:39 (1787382339) [ 2314.148096] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:05:46 (1787382346) [ 2321.308400] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:05:53 (1787382353) [ 2328.252982] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:06:00 (1787382360) [ 2335.707968] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:06:07 (1787382367) [ 2343.206870] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:06:15 (1787382375) [ 2350.946825] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:06:23 (1787382383) [ 2357.984556] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:06:29 (1787382389) [ 2365.545264] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:06:37 (1787382397) [ 2373.421619] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:06:45 (1787382405) [ 2382.078327] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:06:53 (1787382413) [ 2389.804289] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:07:01 (1787382421) [ 2396.994261] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:07:08 (1787382428) [ 2406.061754] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:07:17 (1787382437) [ 2406.248525] LNet: 132287:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2406.248525] mask usage: [+|-] ... [ 2406.433436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2406.452189] alg: No test for adler32 (adler32-zlib) [ 2407.329653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.431687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2407.457257] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2408.639450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2408.669417] LNet: 132464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2408.675451] LNetError: 132464:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2408.685855] LNet: Removed LNI 192.168.204.11@tcp [ 2408.711815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2408.747994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2408.772588] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2416.771829] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:07:28 (1787382448) [ 2425.533695] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:07:37 (1787382457) [ 2435.388649] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:07:46 (1787382466) [ 2444.216822] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:07:55 (1787382475) [ 2451.985152] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:08:03 (1787382483) [ 2459.122541] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:08:11 (1787382491) [ 2465.928618] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:08:17 (1787382497) [ 2473.203333] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:08:24 (1787382504) [ 2480.955575] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:08:32 (1787382512) [ 2487.955971] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:08:39 (1787382519) [ 2495.777399] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:08:47 (1787382527) [ 2503.801348] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:08:55 (1787382535) [ 2512.527505] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2268 sec ========== 03:09:03 (1787382543) [ 2513.030862] LNet: 139932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.039693] LNetError: 139932:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.069134] LNet: Removed LNI 192.168.204.11@tcp [ 2514.086662] Key type .llcrypt unregistered [ 2514.091451] Key type ._llcrypt unregistered [ 2527.695350] Key type ._llcrypt registered [ 2527.697540] Key type .llcrypt registered [ 2527.790296] LNet: 140386:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2527.790296] mask usage: [+|-] ... [ 2528.055462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2528.069700] alg: No test for adler32 (adler32-zlib) [ 2529.187711] Lustre: Lustre: Build Version: 2.17.54_116_gc06913e [ 2529.420164] LNet: Added LNI 192.168.204.11@tcp [8/256/0/180] [ 2531.143177] Key type lgssc registered [ 2532.256794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2598.338116] Lustre: Mounted lustre-client [ 2598.439173] LNet: 141879:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2598.439173] mask usage: [+|-] ... [ 2603.962482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2613.381899] Lustre: Unmounted lustre-client [ 2639.987566] Key type lgssc unregistered [ 2640.277811] LNet: 142861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2640.285457] LNetError: 142861:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2640.308555] LNet: Removed LNI 192.168.204.11@tcp [ 2641.161471] Key type .llcrypt unregistered [ 2641.171598] Key type ._llcrypt unregistered