[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 440916363 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003410] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.006013] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.009035] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011176] LSM: Security Framework initializing [ 0.012074] Yama: becoming mindful. [ 0.013053] SELinux: Initializing. [ 0.014144] *** VALIDATE selinux *** [ 0.023387] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028207] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029216] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030135] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032068] *** VALIDATE tmpfs *** [ 0.033616] *** VALIDATE proc *** [ 0.034334] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.036327] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037203] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039041] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.043217] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.046236] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047818] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048033] ... version: 2 [ 0.049017] ... bit width: 48 [ 0.050018] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053023] ... fixed-purpose events: 3 [ 0.054022] ... event mask: 000000070000000f [ 0.055401] rcu: Hierarchical SRCU implementation. [ 0.057777] smp: Bringing up secondary CPUs ... [ 0.058717] x86: Booting SMP configuration: [ 0.059039] .... node #0, CPUs: #1 #2 #3 [ 0.063115] smp: Brought up 1 node, 4 CPUs [ 0.065021] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.237402] node 0 deferred pages initialised in 167ms [ 0.239142] devtmpfs: initialized [ 0.240245] x86/mm: Memory block size: 128MB [ 0.242390] gcov: version magic: 0x41383552 [ 0.244228] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.245123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246312] pinctrl core: initialized pinctrl subsystem [ 0.247256] [ 0.247879] ************************************************************* [ 0.248020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249015] ** ** [ 0.250019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.251015] ** ** [ 0.252019] ** This means that this kernel is built to expose internal ** [ 0.253015] ** IOMMU data structures, which may compromise security on ** [ 0.254016] ** your system. ** [ 0.255013] ** ** [ 0.256016] ** If you see this message and you are not debugging the ** [ 0.257020] ** kernel, report this immediately to your vendor! ** [ 0.258027] ** ** [ 0.259022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.260022] ************************************************************* [ 0.261831] NET: Registered protocol family 16 [ 0.262551] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.263099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265571] cpuidle: using governor menu [ 0.267768] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.269386] PCI: Using configuration type 1 for base access [ 0.271130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.278115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.279032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.280179] cryptd: max_cpu_qlen set to 1000 [ 0.282092] ACPI: Added _OSI(Module Device) [ 0.283022] ACPI: Added _OSI(Processor Device) [ 0.284020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.285023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.289846] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.292866] ACPI: Interpreter enabled [ 0.293084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.294022] ACPI: Using IOAPIC for interrupt routing [ 0.295165] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.296468] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.306148] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.307079] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.308029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.309116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.311508] acpiphp: Slot [2] registered [ 0.312255] acpiphp: Slot [5] registered [ 0.313174] acpiphp: Slot [6] registered [ 0.314202] acpiphp: Slot [3] registered [ 0.315133] acpiphp: Slot [4] registered [ 0.316135] acpiphp: Slot [7] registered [ 0.317131] acpiphp: Slot [8] registered [ 0.318191] acpiphp: Slot [9] registered [ 0.319135] acpiphp: Slot [10] registered [ 0.320146] acpiphp: Slot [11] registered [ 0.321144] acpiphp: Slot [12] registered [ 0.322180] acpiphp: Slot [13] registered [ 0.323141] acpiphp: Slot [14] registered [ 0.324143] acpiphp: Slot [15] registered [ 0.325198] acpiphp: Slot [16] registered [ 0.326150] acpiphp: Slot [17] registered [ 0.327143] acpiphp: Slot [18] registered [ 0.328140] acpiphp: Slot [19] registered [ 0.329132] acpiphp: Slot [20] registered [ 0.330143] acpiphp: Slot [21] registered [ 0.331189] acpiphp: Slot [22] registered [ 0.332119] acpiphp: Slot [23] registered [ 0.333155] acpiphp: Slot [24] registered [ 0.334145] acpiphp: Slot [25] registered [ 0.335139] acpiphp: Slot [26] registered [ 0.336135] acpiphp: Slot [27] registered [ 0.337176] acpiphp: Slot [28] registered [ 0.338176] acpiphp: Slot [29] registered [ 0.339141] acpiphp: Slot [30] registered [ 0.340165] acpiphp: Slot [31] registered [ 0.341085] PCI host bridge to bus 0000:00 [ 0.342034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.343043] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.344039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.345048] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.346040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.347045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.348230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350053] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352447] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.356935] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.359051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.362031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363683] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364872] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366987] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368930] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.373976] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379549] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.392033] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.398055] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.404025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.411025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.418908] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419529] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.420479] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.421455] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.422270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.425352] iommu: Default domain type: Passthrough [ 0.428745] SCSI subsystem initialized [ 0.430245] ACPI: bus type USB registered [ 0.431132] usbcore: registered new interface driver usbfs [ 0.433095] usbcore: registered new interface driver hub [ 0.435114] usbcore: registered new device driver usb [ 0.437199] pps_core: LinuxPPS API ver. 1 registered [ 0.439018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.443103] PTP clock support registered [ 0.445074] EDAC MC: Ver: 3.0.0 [ 0.446481] PCI: Using ACPI for IRQ routing [ 0.448824] NetLabel: Initializing [ 0.450062] NetLabel: domain hash size = 128 [ 0.451008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.453090] NetLabel: unlabeled traffic allowed by default [ 0.456133] vgaarb: loaded [ 0.457287] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.459010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.465803] clocksource: Switched to clocksource kvm-clock [ 0.570262] VFS: Disk quotas dquot_6.6.0 [ 0.572084] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.573905] *** VALIDATE ramfs *** [ 0.575328] *** VALIDATE hugetlbfs *** [ 0.577100] pnp: PnP ACPI init [ 0.579873] pnp: PnP ACPI: found 6 devices [ 0.596143] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.598848] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.600847] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.602629] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.603837] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.605066] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.606641] NET: Registered protocol family 2 [ 0.608235] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.612461] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.614539] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.618377] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.621704] TCP: Hash tables configured (established 65536 bind 65536) [ 0.623504] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.625586] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.627683] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.629625] NET: Registered protocol family 1 [ 0.631555] RPC: Registered named UNIX socket transport module. [ 0.634037] RPC: Registered udp transport module. [ 0.636033] RPC: Registered tcp transport module. [ 0.638024] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.641775] NET: Registered protocol family 44 [ 0.643520] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.645491] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.647688] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.649560] PCI: CLS 0 bytes, default 64 [ 0.650820] Unpacking initramfs... [ 2.096699] debug: unmapping init [mem 0xffff8a0bbcc64000-0xffff8a0bbffcffff] [ 2.103720] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.106339] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.111027] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.681539] Initialise system trusted keyrings [ 2.682763] Key type blacklist registered [ 2.684296] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.692937] zbud: loaded [ 2.695831] *** VALIDATE nfs *** [ 2.696923] *** VALIDATE nfs4 *** [ 2.698111] pstore: using deflate compression [ 2.701921] Platform Keyring initialized [ 2.818850] NET: Registered protocol family 38 [ 2.822061] Key type asymmetric registered [ 2.823751] Asymmetric key parser 'x509' registered [ 2.825630] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.828918] io scheduler mq-deadline registered [ 2.830105] io scheduler kyber registered [ 2.831586] io scheduler bfq registered [ 2.833759] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.837318] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.840444] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.843111] ACPI: Power Button [PWRF] [ 2.849542] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.855861] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.868365] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.896389] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.925217] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.931810] Non-volatile memory driver v1.3 [ 2.933607] Linux agpgart interface v0.103 [ 2.968723] virtio_blk virtio1: [vda] 145224 512-byte logical blocks (74.4 MB/70.9 MiB) [ 2.970832] vda: detected capacity change from 0 to 74354688 [ 2.985447] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.988000] vdb: detected capacity change from 0 to 1073741824 [ 2.996064] libphy: Fixed MDIO Bus: probed [ 3.005112] usbcore: registered new interface driver usbserial_generic [ 3.006951] usbserial: USB Serial support registered for generic [ 3.009383] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.014521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.016102] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.018183] mousedev: PS/2 mouse device common for all mice [ 3.020361] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.022459] rtc_cmos 00:05: RTC can wake from S4 [ 3.024945] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.026790] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.031765] rtc_cmos 00:05: registered as rtc0 [ 3.034035] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.037305] intel_pstate: CPU model not supported [ 3.040828] hid: raw HID events driver (C) Jiri Kosina [ 3.043373] usbcore: registered new interface driver usbhid [ 3.045810] usbhid: USB HID core driver [ 3.047730] drop_monitor: Initializing network drop monitor service [ 3.050557] Initializing XFRM netlink socket [ 3.053058] NET: Registered protocol family 10 [ 3.056577] Segment Routing with IPv6 [ 3.058091] NET: Registered protocol family 17 [ 3.060536] mpls_gso: MPLS GSO support [ 3.067198] RAS: Correctable Errors collector initialized. [ 3.069048] AVX version of gcm_enc/dec engaged. [ 3.070420] AES CTR mode by8 optimization enabled [ 3.150816] sched_clock: Marking stable (3150784321, 0)->(4297538877, -1146754556) [ 3.154239] registered taskstats version 1 [ 3.156154] Loading compiled-in X.509 certificates [ 3.158203] zswap: loaded using pool lzo/zbud [ 3.181548] Key type big_key registered [ 3.193124] Key type encrypted registered [ 3.194270] ima: No TPM chip found, activating TPM-bypass! [ 3.195540] ima: Allocated hash algorithm: sha1 [ 3.196719] ima: No architecture policies found [ 3.198627] evm: Initialising EVM extended attributes: [ 3.200628] evm: security.selinux [ 3.201765] evm: security.ima [ 3.202562] evm: security.capability [ 3.203939] evm: HMAC attrs: 0x1 [ 3.206303] rtc_cmos 00:05: setting system clock to 2026-06-23 19:06:20 UTC (1782241580) [ 3.213254] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.216362] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 3.226082] Write protecting the kernel read-only data: 28672k [ 3.230132] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.232888] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 3.263488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.273495] systemd[1]: Detected virtualization kvm. [ 3.275738] systemd[1]: Detected architecture x86-64. [ 3.277935] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.299451] systemd[1]: No hostname configured. [ 3.301119] systemd[1]: Set hostname to . [ 3.303307] random: systemd: uninitialized urandom read (16 bytes read) [ 3.305159] systemd[1]: Initializing machine ID from random generator. [ 3.451270] random: systemd: uninitialized urandom read (16 bytes read) [ 3.454446] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.458472] random: systemd: uninitialized urandom read (16 bytes read) [ 3.460294] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.464796] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ 3.594663] random: fast init done Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.045049] device-mapper: uevent: version 1.0.3 [ 4.046550] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.883974] virtio_net virtio0 ens2: renamed from eth0 [ 4.955789] scsi host0: ata_piix [ 4.970556] scsi host1: ata_piix [ 4.972135] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.974308] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.391572] dracut-initqueue[601]: RTNETLINK answers: File exists [ 9.710838] random: crng init done [ 9.711722] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.199788] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.461398] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.698565] SELinux: Disabled at runtime. [ 11.758756] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.767593] systemd[1]: Detected virtualization kvm. [ 11.769423] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.288294] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.293160] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.297661] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.300553] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.303454] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.314249] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.320889] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ 12.414916] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.826337] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.146654] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.154944] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.244732] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.271594] EDAC sbridge: Ver: 1.1.2 [ 14.701240] Key type dns_resolver registered [ 15.010660] NFS: Registering the id_resolver key type [ 15.012665] Key type id_resolver registered [ 15.014394] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg242-client login: [ 63.174795] libcfs: loading out-of-tree module taints kernel. [ 63.300395] Key type ._llcrypt registered [ 63.304265] Key type .llcrypt registered [ 64.042976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.053943] alg: No test for adler32 (adler32-zlib) [ 65.712946] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 66.726911] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 68.687382] Key type lgssc registered [ 70.881985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 222.103061] Lustre: Mounted lustre-client [ 227.738370] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.458423] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing check_logdir /tmp/testlogs/ [ 247.782426] Lustre: lustre-OST0000-osc-ffff8a0c11056000: disconnect after 24s idle [ 252.742813] Lustre: DEBUG MARKER: oleg242-client.virtnet: executing yml_node [ 259.373278] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 262.698843] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 266.280115] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 268.222219] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 23 15:10:43 EDT 2026 [ 286.367211] Lustre: Unmounted lustre-client [ 320.378844] hrtimer: interrupt took 5599346 ns [ 321.458620] Key type lgssc unregistered [ 322.098939] LNet: 8407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.111544] LNetError: 8407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.160318] LNet: Removed LNI 192.168.202.42@tcp [ 323.329359] Key type .llcrypt unregistered [ 323.331171] Key type ._llcrypt unregistered [ 355.722656] Key type ._llcrypt registered [ 355.726779] Key type .llcrypt registered [ 355.809605] LNet: 8988:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 355.809605] mask usage: [+|-] ... [ 356.102246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 356.126861] alg: No test for adler32 (adler32-zlib) [ 357.058145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 357.175405] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 357.240916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 362.219045] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 362.765679] LNet: 9295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.773783] LNetError: 9295:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 362.797988] LNet: Removed LNI 192.168.202.42@tcp [ 363.672498] Key type .llcrypt unregistered [ 363.676132] Key type ._llcrypt unregistered [ 372.135510] Key type ._llcrypt registered [ 372.138067] Key type .llcrypt registered [ 373.758268] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:12:29 (1782241949) [ 373.923930] LNet: 9663:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 373.923930] mask usage: [+|-] ... [ 374.120714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.152092] alg: No test for adler32 (adler32-zlib) [ 375.139926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.223687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.243796] LNet: 9706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.269403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.355954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.384915] LNet: 9713:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 382.771429] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:12:38 (1782241958) [ 382.951355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.980742] LNet: 10290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.031875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.101496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 390.847712] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:12:46 (1782241966) [ 391.035142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.076435] LNet: 10879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.105063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.183356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 398.452722] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:12:54 (1782241974) [ 398.652357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.676272] LNet: 11459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.711716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.816692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 407.016555] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:13:02 (1782241982) [ 407.192430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.227346] LNet: 12043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.267842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.368541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 407.768979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 414.722473] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:13:10 (1782241990) [ 414.808967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.879101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.037453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 415.512466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 415.954105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 423.065358] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:13:18 (1782241998) [ 423.225983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.264725] LNet: 13235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.269390] LNet: 13235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 423.286494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.363596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 429.550797] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:13:25 (1782242005) [ 429.906833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.968312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.061074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 430.085402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 430.616651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 430.654922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 431.074686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 431.107848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.591289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 431.624283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 432.102670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 432.168343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.641710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 432.670237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.102810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 433.175687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 433.680784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 433.718313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.164348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 434.205495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 434.754913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 434.777191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 435.402444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 435.442991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 436.135718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 436.174530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 436.603070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 436.620996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 443.200715] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:13:38 (1782242018) [ 443.446678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.459329] LNet: 14501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.463075] LNet: 14501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 443.474518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.530398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 443.560775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 450.856286] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:13:46 (1782242026) [ 451.121104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.167233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.268166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 451.323672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 458.894107] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:13:54 (1782242034) [ 459.083939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.130411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.258766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 459.299416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 467.394849] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:14:03 (1782242043) [ 467.532641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.593521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.685956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 467.720767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 475.170546] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:14:11 (1782242051) [ 475.300784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.353930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.495704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 475.546945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 483.280273] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:14:19 (1782242059) [ 483.475657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.496570] LNet: 17433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.503210] LNet: 17433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 483.521819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.656440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 483.737871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 491.490806] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:14:27 (1782242067) [ 491.762519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.798515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.968854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.740962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.107643] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:14:36 (1782242076) [ 501.232486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 501.289415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.541978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.814586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.861843] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:14:45 (1782242085) [ 510.312259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.392207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.577772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.992698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.917510] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:14:54 (1782242094) [ 519.365458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.418454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.975309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.217824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.871883] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:15:03 (1782242103) [ 528.002646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.040675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.149943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.203877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.793978] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:15:12 (1782242112) [ 537.005887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.096866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.247602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.354740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 545.338699] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:15:21 (1782242121) [ 545.501150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.577528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.679611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 553.319875] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:15:29 (1782242129) [ 553.638895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.673519] LNet: 22288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.682895] LNet: 22288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 553.725920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.844321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 561.109381] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:15:36 (1782242136) [ 561.363302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.419549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.498984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 562.068381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 569.602861] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:15:45 (1782242145) [ 569.723710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.762572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.906516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 569.954355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 577.498432] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:15:53 (1782242153) [ 577.756964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.821373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.947781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 578.012679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 585.128251] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:16:01 (1782242161) [ 585.444900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.503967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.654179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 585.705058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 594.418814] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:16:09 (1782242169) [ 594.578611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.625076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.814223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 594.870095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 602.626872] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:16:18 (1782242178) [ 602.812589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.882924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.970645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 603.041165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 603.068933] LNetError: 25822:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 603.115776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 603.168304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 610.765364] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:16:26 (1782242186) [ 610.898467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.008647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.120989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.m4KViGmrOV [ 611.241937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.m4KViGmrOV [ 611.304577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.m4KViGmrOV [ 618.614133] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:16:34 (1782242194) [ 618.904958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.948826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.994615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 619.025297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 619.095082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 625.731947] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:16:41 (1782242201) [ 625.879531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.898465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.939595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.016667] LNet: Message forwarding will be enabled in 2s [ 631.138420] LNet: Message forwarding enabled [ 631.533468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 631.556481] LNet: Message forwarding will stop in 2s to 6s [ 636.140571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 636.322284] LNet: Message forwarding disabled [ 638.634606] LNet: Message forwarding will be enabled in 2s [ 642.465333] LNet: Message forwarding enabled [ 646.254567] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:17:02 (1782242222) [ 646.615312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.145896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.230092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 655.388243] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:17:10 (1782242230) [ 655.566211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.630738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.711957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 655.752399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 655.788889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 655.841837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 655.899426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 655.938827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 655.972924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.017955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 656.057957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 656.094362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 656.130821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 656.176652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 656.231204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 656.271322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 656.324950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 656.366217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 656.398412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 656.436421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.479667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 656.502704] LNetError: 28893:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 664.170730] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:17:19 (1782242239) [ 664.271456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.346341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.470823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.486514] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:17:27 (1782242247) [ 675.226823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg242-server [ 677.733965] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:17:33 (1782242253) [ 677.849254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.899057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.972954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.037506] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 680.182682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[43-47]@tcp [ 680.216549] LNet: 30612:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 680.306708] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.46@tcp is being used as a gateway but routing feature is not turned on [ 680.345434] LNetError: 30399:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.46@tcp has gone from up to down [ 681.315125] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.44@tcp is being used as a gateway but routing feature is not turned on [ 681.327303] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 682.342604] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.43@tcp is being used as a gateway but routing feature is not turned on [ 682.356218] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 684.405384] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 684.416893] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 687.142251] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:17:43 (1782242263) [ 688.495406] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.45@tcp is being used as a gateway but routing feature is not turned on [ 688.508313] LNetError: 30399:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 689.390444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg242-server [ 691.343410] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:17:47 (1782242267) [ 691.479780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.515222] LNet: 31542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.521367] LNet: 31542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 691.542598] LNetError: 31542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.582077] LNet: Removed LNI 192.168.202.42@tcp [ 692.663681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.779679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 692.815852] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 693.534197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 693.552111] LNet: 31666:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 693.558927] LNet: 31666:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 693.575638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[43-48/2]@tcp [ 693.612126] LNetError: 31553:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.43@tcp has gone from up to down [ 693.636487] LNetError: 31553:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 700.060374] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:17:56 (1782242276) [ 700.347641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.365954] LNetError: 32259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.422158] LNet: Removed LNI 192.168.202.42@tcp [ 701.483394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.604852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 701.663877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.728704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.833448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.924292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.970710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.064871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 702.210140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.288686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.384474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.520525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.566566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.616144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 702.723539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.805256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.883738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.016767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.081348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.181541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 703.324854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.414049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.505568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.653110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.726657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.819893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 703.952195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.038437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.184500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 704.294204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.341526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.401813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 704.495118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.547170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.657574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 711.785951] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:18:07 (1782242287) [ 713.476190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg242-server [ 715.820396] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:18:11 (1782242291) [ 718.196691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg242-server [ 719.961263] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:18:15 (1782242295) [ 720.832887] Key type .llcrypt unregistered [ 720.835406] Key type ._llcrypt unregistered [ 727.645676] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 727.897289] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 727.922215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 727.941104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 730.594314] Key type ._llcrypt registered [ 730.596288] Key type .llcrypt registered [ 730.621404] LNet: 33941:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 730.621404] mask usage: [+|-] ... [ 730.739615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.756989] alg: No test for adler32 (adler32-zlib) [ 731.729212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.793871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.817848] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 732.461906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.496761] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.881212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 734.909772] LNet: Removed LNI 10.1.2.3@tcp [ 735.022597] LNet: 34383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.029632] LNetError: 34383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.066286] LNet: Removed LNI 192.168.202.42@tcp [ 735.839555] Key type .llcrypt unregistered [ 735.846477] Key type ._llcrypt unregistered [ 748.455853] Key type ._llcrypt registered [ 748.457161] Key type .llcrypt registered [ 749.928703] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:18:45 (1782242325) [ 750.695418] Key type .llcrypt unregistered [ 750.696773] Key type ._llcrypt unregistered [ 757.931563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 758.056980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 758.067982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 758.077806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 760.885043] Key type ._llcrypt registered [ 760.887096] Key type .llcrypt registered [ 760.948814] LNet: 35419:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 760.948814] mask usage: [+|-] ... [ 761.200617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 761.237778] alg: No test for adler32 (adler32-zlib) [ 762.208698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.254099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.275578] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 762.895130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.926261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.722984] LNet: 35876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.727700] LNetError: 35876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 764.733133] LNet: Removed LNI 10.1.2.3@tcp [ 764.754503] LNet: Removed LNI 192.168.202.42@tcp [ 765.405496] Key type .llcrypt unregistered [ 765.411381] Key type ._llcrypt unregistered [ 778.199657] Key type ._llcrypt registered [ 778.202991] Key type .llcrypt registered [ 779.497274] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:19:15 (1782242355) [ 780.274602] Key type .llcrypt unregistered [ 780.276702] Key type ._llcrypt unregistered [ 785.953223] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 786.186240] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 786.194267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 786.198108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.909209] Key type ._llcrypt registered [ 788.914709] Key type .llcrypt registered [ 788.966996] LNet: 36916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 788.966996] mask usage: [+|-] ... [ 789.144473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.159925] alg: No test for adler32 (adler32-zlib) [ 790.104018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.161135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 790.192498] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 792.326499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.371462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.785324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 794.800161] LNet: Removed LNI 10.1.2.3@tcp [ 794.809799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 794.835855] LNet: Removed LNI 10.1.2.31@tcp [ 794.846385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 794.863539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.537572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.562447] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 796.772945] LNet: 37800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.781770] LNetError: 37800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 796.788488] LNet: Removed LNI 10.1.2.31@tcp [ 796.797390] LNet: Removed LNI 10.1.2.3@tcp [ 797.552463] Key type .llcrypt unregistered [ 797.554980] Key type ._llcrypt unregistered [ 814.321918] Key type ._llcrypt registered [ 814.324418] Key type .llcrypt registered [ 815.739765] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:19:51 (1782242391) [ 816.041219] LNet: 38593:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 816.041219] mask usage: [+|-] ... [ 816.181303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 816.191157] alg: No test for adler32 (adler32-zlib) [ 817.209136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.273727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 817.294076] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 817.980836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.999303] LNet: 38754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.008077] LNetError: 38754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.031000] LNet: Removed LNI 192.168.202.42@tcp [ 818.048524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.087836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.42@tcp [ 818.104954] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 818.814834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.42@tcp [ 818.852347] LNet: Removed LNI 192.168.202.42@tcp [ 818.905542] LNet: 38889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.747548] Key type .llcrypt unregistered [ 819.749308] Key type ._llcrypt unregistered [ 831.365894] Key type ._llcrypt registered [ 831.369730] Key type .llcrypt registered [ 832.577801] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:20:08 (1782242408) [ 834.142354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg242-server [ 835.572852] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:20:11 (1782242411) [ 836.015452] Key type .llcrypt unregistered [ 836.019584] Key type ._llcrypt unregistered [ 841.753667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 841.924766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 841.928260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 841.931553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 844.515685] Key type ._llcrypt registered [ 844.517455] Key type .llcrypt registered [ 844.544837] LNet: 40265:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 844.544837] mask usage: [+|-] ... [ 844.764412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 844.799202] alg: No test for adler32 (adler32-zlib) [ 845.733557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.787789] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 846.605674] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 848.637970] LNet: 40725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.645861] LNetError: 40725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.651159] LNet: Removed LNI 192.168.202.42@tcp1 [ 848.667852] LNet: Removed LNI 10.1.2.3@tcp2 [ 849.263656] Key type .llcrypt unregistered [ 849.267342] Key type ._llcrypt unregistered [ 860.684299] Key type ._llcrypt registered [ 860.689119] Key type .llcrypt registered [ 862.065857] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:20:37 (1782242437) [ 862.349890] LNet: 41529:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 862.349890] mask usage: [+|-] ... [ 862.500211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.547774] alg: No test for adler32 (adler32-zlib) [ 863.505276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.544537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.564575] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 864.351401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.374760] LNet: 41694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.380685] LNetError: 41694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.398613] LNet: Removed LNI 192.168.202.42@tcp [ 864.432558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.471686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 864.487055] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 871.521846] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:20:47 (1782242447) [ 871.645595] LNet: 42277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.654827] LNetError: 42277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.680552] LNet: Removed LNI 192.168.202.42@tcp [ 872.424111] Key type .llcrypt unregistered [ 872.426520] Key type ._llcrypt unregistered [ 878.614763] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.753221] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.760792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.765936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.261848] Key type ._llcrypt registered [ 881.264398] Key type .llcrypt registered [ 881.294928] LNet: 42557:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 881.294928] mask usage: [+|-] ... [ 881.499424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.517620] alg: No test for adler32 (adler32-zlib) [ 882.454343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.514233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.556969] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 883.396491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 883.423525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.378887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.392884] LNet: 43023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.402359] LNetError: 43023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.406425] LNet: Removed LNI 10.1.2.3@tcp [ 885.425470] LNet: Removed LNI 192.168.202.42@tcp [ 885.453310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.542089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.564356] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 885.576026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.013866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.032760] LNet: 43050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.038191] LNetError: 43050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 886.044611] LNetError: 43050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 886.053572] LNet: Removed LNI 10.1.2.3@tcp [ 886.067304] LNet: Removed LNI 192.168.202.42@tcp [ 886.088783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.156847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 886.195666] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 886.200092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.839724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 886.860995] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 887.710244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.720664] LNet: 43212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.725789] LNetError: 43212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 887.729276] LNetError: 43212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.732277] LNet: Removed LNI 10.1.2.3@tcp [ 887.733720] LNet: Removed LNI 192.168.202.42@tcp [ 887.753146] LNet: Removed LNI 192.168.202.42@tcp2 [ 887.766449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.811904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 887.838844] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 887.842686] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 887.846497] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 894.848784] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:21:10 (1782242470) [ 895.112264] LNet: 43810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.141245] LNetError: 43810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.161774] LNetError: 43810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 895.173895] LNet: Removed LNI 10.1.2.3@tcp [ 895.179373] LNet: Removed LNI 192.168.202.42@tcp [ 895.203472] LNet: Removed LNI 192.168.202.42@tcp2 [ 895.994769] Key type .llcrypt unregistered [ 895.996800] Key type ._llcrypt unregistered [ 902.866372] Key type ._llcrypt registered [ 902.868929] Key type .llcrypt registered [ 902.895594] LNet: 44035:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 902.895594] mask usage: [+|-] ... [ 903.019666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.039435] alg: No test for adler32 (adler32-zlib) [ 904.124752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.184458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.216449] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 905.162403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.181610] LNet: 44205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.187144] LNetError: 44205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.198491] LNet: Removed LNI 192.168.202.42@tcp [ 905.209920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.259824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 905.289031] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 905.790480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.822310] LNet: 44235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.831455] LNetError: 44235:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.854708] LNet: Removed LNI 192.168.202.42@tcp [ 905.895558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.070988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 906.151883] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 907.011310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.036430] LNet: 44265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.047290] LNetError: 44265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.065786] LNet: Removed LNI 192.168.202.42@tcp [ 907.098849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.184968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 907.228679] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 907.815417] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 907.927238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 907.939780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 907.942188] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.445556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.464363] LNet: 44341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.468334] LNetError: 44341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.479085] LNet: Removed LNI 192.168.202.42@tcp [ 909.507698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.556865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.579029] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 917.391152] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:21:33 (1782242493) [ 917.851447] LNet: 44937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.862904] LNetError: 44937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.899317] LNet: Removed LNI 192.168.202.42@tcp [ 918.846068] Key type .llcrypt unregistered [ 918.853685] Key type ._llcrypt unregistered [ 927.950670] Key type ._llcrypt registered [ 927.959853] Key type .llcrypt registered [ 928.064835] LNet: 45162:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 928.064835] mask usage: [+|-] ... [ 928.451641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.514977] alg: No test for adler32 (adler32-zlib) [ 929.768949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.897511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.990650] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 931.640987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.678109] LNet: 45336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.697083] LNetError: 45336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.718196] LNet: Removed LNI 192.168.202.42@tcp [ 931.770456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.919488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 932.011757] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 940.823545] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:21:56 (1782242516) [ 941.169164] LNet: 45929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.177939] LNetError: 45929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.203583] LNet: Removed LNI 192.168.202.42@tcp [ 941.830828] Key type .llcrypt unregistered [ 941.833140] Key type ._llcrypt unregistered [ 948.976430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.291385] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.293867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.296616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.904434] Key type ._llcrypt registered [ 951.906631] Key type .llcrypt registered [ 951.946632] LNet: 46206:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 951.946632] mask usage: [+|-] ... [ 952.094908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.111918] alg: No test for adler32 (adler32-zlib) [ 953.147691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.194465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.235287] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 954.290807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 954.309090] LNet: 46380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.312233] LNetError: 46380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.322402] LNet: Removed LNI 192.168.202.42@tcp [ 954.336110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.405816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 954.429417] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 962.544583] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:22:17 (1782242537) [ 962.679348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.729138] LNet: 46973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.748763] LNetError: 46973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.795194] LNet: Removed LNI 192.168.202.42@tcp [ 962.833292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.960358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 969.313704] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:22:25 (1782242545) [ 969.640336] LNet: 47561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.101523] Key type .llcrypt unregistered [ 970.103696] Key type ._llcrypt unregistered [ 976.290205] Key type ._llcrypt registered [ 976.299607] Key type .llcrypt registered [ 976.396622] LNet: 47786:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 976.396622] mask usage: [+|-] ... [ 976.703650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.714797] alg: No test for adler32 (adler32-zlib) [ 977.734953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.778692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.790967] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 978.697673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.730387] LNet: 47958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.736831] LNetError: 47958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.755583] LNet: Removed LNI 192.168.202.42@tcp [ 978.800312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.895847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 978.922031] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 985.292546] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:22:41 (1782242561) [ 985.569679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.585646] LNet: 48552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.590618] LNetError: 48552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.603877] LNet: Removed LNI 192.168.202.42@tcp [ 985.622497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.703621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 992.836306] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:22:48 (1782242568) [ 995.171150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 997.035850] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:22:52 (1782242572) [ 998.527460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1000.568933] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:22:56 (1782242576) [ 1002.448573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1003.940387] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:23:00 (1782242580) [ 1005.758158] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1007.486641] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:23:03 (1782242583) [ 1009.025539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1010.767620] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:23:06 (1782242586) [ 1011.073840] LNet: 50604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.697509] Key type .llcrypt unregistered [ 1011.699518] Key type ._llcrypt unregistered [ 1018.326415] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.579567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.590395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.598755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.114530] Key type ._llcrypt registered [ 1021.120312] Key type .llcrypt registered [ 1021.158558] LNet: 50888:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1021.158558] mask usage: [+|-] ... [ 1021.311366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.325508] alg: No test for adler32 (adler32-zlib) [ 1022.243423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.313925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.332342] LNet: Added LNI 192.168.202.42@tcp [8/512/0/180] [ 1022.337443] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1022.340463] LNet: 50927:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1022.424692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.445248] LNet: 50946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.450917] LNetError: 50946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.457096] LNet: Removed LNI 10.1.2.3@tcp [ 1022.459551] LNetError: 50946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.479280] LNet: Removed LNI 192.168.202.42@tcp [ 1022.515484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.589195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.606728] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1022.665241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.688822] LNet: 50970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.694685] LNetError: 50970:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.711212] LNet: Removed LNI 192.168.202.42@tcp [ 1022.736567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.803182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1022.822444] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1022.827511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1022.831371] LNet: 50978:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1022.920211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.943708] LNet: 50996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.948751] LNetError: 50996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1022.953374] LNet: Removed LNI 10.1.2.3@tcp [ 1022.955566] LNetError: 50996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.965578] LNet: Removed LNI 192.168.202.42@tcp [ 1022.985623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.042453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.059653] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1023.064026] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.068219] LNet: 51004:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.223188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.241825] LNet: 51030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.247620] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.254817] LNet: Removed LNI 10.1.2.3@tcp [ 1023.257427] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.269781] LNet: Removed LNI 192.168.202.42@tcp [ 1023.290762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.367806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.383703] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 1023.388055] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1023.478725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.494414] LNet: 51058:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.501456] LNetError: 51058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1023.508657] LNet: Removed LNI 10.1.2.3@tcp [ 1023.512316] LNetError: 51058:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.527167] LNet: Removed LNI 192.168.202.42@tcp [ 1023.543372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.594689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1023.613378] LNet: Added LNI 192.168.202.42@tcp [8/256/0/182] [ 1023.617559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1023.620631] LNet: 51066:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1023.684619] LNet: Removed LNI 10.1.2.3@tcp [ 1023.698723] LNet: Removed LNI 192.168.202.42@tcp [ 1024.371453] Key type .llcrypt unregistered [ 1024.373346] Key type ._llcrypt unregistered [ 1035.390304] Key type ._llcrypt registered [ 1035.393435] Key type .llcrypt registered [ 1036.906459] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:23:32 (1782242612) [ 1037.243412] LNet: 51880:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1037.243412] mask usage: [+|-] ... [ 1037.452315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.475710] alg: No test for adler32 (adler32-zlib) [ 1038.527332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.586470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1038.613594] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1039.589505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.42 [ 1039.627547] LNet: 52041:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1039.647398] LNet: Removed LNI 192.168.202.42@tcp [ 1046.755775] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:23:42 (1782242622) [ 1046.852735] LNet: 52603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.862645] LNetError: 52603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.890535] LNet: Removed LNI 192.168.202.42@tcp [ 1047.510310] Key type .llcrypt unregistered [ 1047.511704] Key type ._llcrypt unregistered [ 1055.743892] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.111938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.895844] Key type ._llcrypt registered [ 1058.898603] Key type .llcrypt registered [ 1058.932326] LNet: 52893:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1058.932326] mask usage: [+|-] ... [ 1059.066586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.076094] alg: No test for adler32 (adler32-zlib) [ 1060.686260] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1068.227834] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:24:03 (1782242643) [ 1068.341352] LNet: 53519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.355062] LNetError: 53519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1068.392241] LNet: Removed LNI 10.1.2.3@tcp [ 1069.248600] Key type .llcrypt unregistered [ 1069.252224] Key type ._llcrypt unregistered [ 1076.337535] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1076.602440] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1076.606971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1076.612715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.374237] Key type ._llcrypt registered [ 1079.377313] Key type .llcrypt registered [ 1079.408712] LNet: 53797:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1079.408712] mask usage: [+|-] ... [ 1079.514187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.524784] alg: No test for adler32 (adler32-zlib) [ 1080.537303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1087.261645] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:24:23 (1782242663) [ 1089.203776] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1090.651378] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:24:26 (1782242666) [ 1091.014912] LNet: 54708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.029932] LNetError: 54708:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1091.051835] LNet: Removed LNI 10.1.2.3@tcp [ 1091.645538] Key type .llcrypt unregistered [ 1091.653936] Key type ._llcrypt unregistered [ 1098.388799] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1098.736112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1098.742773] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1098.754736] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1101.571605] Key type ._llcrypt registered [ 1101.574603] Key type .llcrypt registered [ 1101.618435] LNet: 54982:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1101.618435] mask usage: [+|-] ... [ 1101.816796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.828526] alg: No test for adler32 (adler32-zlib) [ 1102.807798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.905408] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1111.424801] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:24:47 (1782242687) [ 1111.867881] LNet: 55939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.872570] LNetError: 55939:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1111.884930] LNet: Removed LNI 10.1.2.3@tcp [ 1112.450897] Key type .llcrypt unregistered [ 1112.456393] Key type ._llcrypt unregistered [ 1120.189664] Key type ._llcrypt registered [ 1120.196254] Key type .llcrypt registered [ 1120.236378] LNet: 56163:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1120.236378] mask usage: [+|-] ... [ 1120.576668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.622953] alg: No test for adler32 (adler32-zlib) [ 1121.673318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.728326] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1129.585492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1130.300117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1130.347599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1130.396332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1130.457534] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1130.719452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.743115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1130.777675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.808674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.845380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.895153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.952756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.975114] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1130.987573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.006639] LNet: 56378:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.026303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.055760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.098182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.153667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.205418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1131.260746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1131.310807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1131.753356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1131.788292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1131.828954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1131.845373] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1132.128973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.161327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1132.205785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.240246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.273701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.303684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.339082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.366489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.387702] LNet: 56487:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.406545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.436952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.463384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.487173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.517115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.541927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.571657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1133.157520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1133.186481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1133.212887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1133.227916] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1133.234367] LNet: Skipped 1 previous similar message [ 1133.426374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.451221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1133.477388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.498515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.523843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.544367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.566577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.589580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.603327] LNet: 56596:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.627618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.664434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.708397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.765973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.819193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.859971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.894587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.382098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1134.417288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1134.456231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1134.649727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.675872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1134.699499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.726261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.752808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.778879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.811713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.840276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.868500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.896190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.922493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.953384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.983932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.015523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.048054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1135.598917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1135.660468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1135.717891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1135.732643] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1135.740165] LNet: Skipped 1 previous similar message [ 1135.983857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1136.052764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1136.083380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1136.109782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.141160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.186866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.231856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.272118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.307890] LNet: 56815:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.328364] LNet: 56815:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1136.353268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.401321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.461680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.487982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.519494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.544540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.578884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1137.091690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1137.133879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1137.184785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1137.401101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.432619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1137.466101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.502411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.527703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.556563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.589890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.621747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.649392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.674277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.696626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.716524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.732807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.766708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.804582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.150185] LNet: 56965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.156651] LNetError: 56965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.183849] LNet: Removed LNI 192.168.202.42@tcp [ 1139.679402] Key type .llcrypt unregistered [ 1139.685344] Key type ._llcrypt unregistered [ 1158.042596] Key type ._llcrypt registered [ 1158.044513] Key type .llcrypt registered [ 1159.243204] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:25:35 (1782242735) [ 1159.971510] Key type .llcrypt unregistered [ 1159.973163] Key type ._llcrypt unregistered [ 1166.234234] Key type ._llcrypt registered [ 1166.237577] Key type .llcrypt registered [ 1166.305515] LNet: 58004:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1166.305515] mask usage: [+|-] ... [ 1166.579676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.603951] alg: No test for adler32 (adler32-zlib) [ 1167.588805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1167.611601] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1176.447097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1187.846828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.889828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1187.938062] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1189.330477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1189.363359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1189.394708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1189.432417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1189.475976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 1 [ 1189.510767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 1 [ 1189.538281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 1 [ 1189.561692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 1 [ 1189.606276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1189.626316] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1189.632934] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1189.875506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.896079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1189.927354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.953509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.984460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.010955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.048038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.078967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.094617] LNet: 58434:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.111516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.146748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.181847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.211653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.241453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.273463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.302487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.873851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1190.913884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1190.946339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1190.981440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1191.008967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 1 [ 1191.044850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 1 [ 1191.071229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 1 [ 1191.097834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 1 [ 1191.124315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1191.137515] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1191.139769] LNet: Skipped 2 previous similar messages [ 1191.321707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.351797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1191.384850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.420607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.446495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.468432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.485774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.512740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.532804] LNet: 58556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.563844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.603813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.629593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.652434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.678649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.705987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.738282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.270908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1192.303701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1192.332282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1192.358402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1192.382763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 1 [ 1192.408408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 1 [ 1192.434956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 1 [ 1192.464315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 1 [ 1192.490438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1192.503910] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1192.507935] LNet: Skipped 2 previous similar messages [ 1192.705528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.757562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.784263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.824844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.866790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.890872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.922492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.943965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.958845] LNet: 58677:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.975470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.003609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.060746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.116402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.149465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.168054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.214987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.769621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1193.804452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1193.835445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1193.874638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1193.905688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 1 [ 1193.935325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 1 [ 1193.976157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 1 [ 1194.018289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 1 [ 1194.072847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1194.320631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.344476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1194.386982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.428670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.517819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.557905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.611667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.639325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.668303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.696160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.737846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.766977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.804163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.852925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.901149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.465343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1195.488803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1195.560241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1195.619394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1195.658981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 1 [ 1195.706353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 1 [ 1195.771388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 1 [ 1195.840792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 1 [ 1195.919474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1195.944247] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1195.950155] LNet: Skipped 5 previous similar messages [ 1196.173571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.198067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.223394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.248660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.270906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.294124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.323621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.345809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.369630] LNet: 58920:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.374842] LNet: 58920:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1196.397199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.444376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.487669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.532234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.590460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.616979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.647696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.207099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1197.239442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1197.282909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1197.323137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1197.379410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_error -r 1 [ 1197.436310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_error -r 1 [ 1197.496168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_error -r 1 [ 1197.577424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_error -r 1 [ 1197.676175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1198.015677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.064742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1198.090404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.149358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.184956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.222114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.267762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.310796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.342363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1198.376641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.435360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.465436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.520743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.592854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.663127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1204.691613] LNet: 59117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.707101] LNetError: 59117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.710896] LNet: Removed LNI 192.168.202.42@tcp [ 1204.729795] LNet: Removed LNI 192.168.202.42@tcp1 [ 1205.305574] Key type .llcrypt unregistered [ 1205.307244] Key type ._llcrypt unregistered [ 1223.147255] Key type ._llcrypt registered [ 1223.149544] Key type .llcrypt registered [ 1224.414068] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:26:40 (1782242800) [ 1225.478549] Key type .llcrypt unregistered [ 1225.481721] Key type ._llcrypt unregistered [ 1232.325629] Key type ._llcrypt registered [ 1232.328551] Key type .llcrypt registered [ 1232.390430] LNet: 60154:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1232.390430] mask usage: [+|-] ... [ 1232.624272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1232.652391] alg: No test for adler32 (adler32-zlib) [ 1233.631812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1233.652655] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1240.547359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1241.290807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1241.324973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1241.355935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1241.376511] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1241.608528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.639231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1241.666971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.696859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.741989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.799872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.841259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.889268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.912624] LNet: 60369:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.941468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.995759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.028990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.076341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.158697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.242527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.327498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.914887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1242.957477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1242.987646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1243.001706] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1243.204618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.220905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1243.246796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.271710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.300915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.342850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.382466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.406776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.422404] LNet: 60478:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.433306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.448687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.473649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.501541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.527354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.555175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.587545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.064675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1244.098280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1244.151918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1244.183768] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1244.423776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.451512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1244.480666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.516773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.568524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.617645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.647940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.681702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.703539] LNet: 60587:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.720472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.759240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.805509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.853834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.879750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.908028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.945444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1245.120242] LNet: 60628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.124736] LNetError: 60628:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.136091] LNet: Removed LNI 192.168.202.42@tcp [ 1245.445542] Key type .llcrypt unregistered [ 1245.447991] Key type ._llcrypt unregistered [ 1263.699850] Key type ._llcrypt registered [ 1263.701708] Key type .llcrypt registered [ 1264.755661] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:27:20 (1782242840) [ 1265.566109] Key type .llcrypt unregistered [ 1265.567538] Key type ._llcrypt unregistered [ 1272.026431] Key type ._llcrypt registered [ 1272.031738] Key type .llcrypt registered [ 1272.073271] LNet: 61666:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1272.073271] mask usage: [+|-] ... [ 1272.353614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.377403] alg: No test for adler32 (adler32-zlib) [ 1273.303679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.329687] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1284.737164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1294.864129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.939517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1294.999318] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1296.396837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1296.462193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1296.538784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1296.623884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1296.663198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 1 [ 1296.725595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 1 [ 1296.844053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 1 [ 1296.900428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 1 [ 1296.973757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1297.014346] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1297.025118] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1297.289209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.332389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1297.375281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.430846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.492808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.569702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.654565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.701489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.730380] LNet: 62097:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.759476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.794782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.837284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.870143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.900655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.946229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1298.015972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1298.073988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.550719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1298.594982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1298.663353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1298.706571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1298.744555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_error -r 1 [ 1298.783978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_error -r 1 [ 1298.844967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_error -r 1 [ 1298.908583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_error -r 1 [ 1298.977440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1299.007911] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1299.020486] LNet: Skipped 2 previous similar messages [ 1299.272497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1299.334777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1299.386441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1299.445777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1299.487255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1299.532100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1299.590647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.658938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.703340] LNet: 62221:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.734962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.793797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.863600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.941341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.997776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.065832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1300.126769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1300.170099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1300.652547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1300.679534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1300.709250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1300.746522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1300.774249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_timeout -r 1 [ 1300.806955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_timeout -r 1 [ 1300.839850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_timeout -r 1 [ 1300.871914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_timeout -r 1 [ 1300.910808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.142@tcp [ 1300.937584] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1301.219829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.268445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1301.323545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.376862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.427028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1301.487556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.532974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.564895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.584781] LNet: 62344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.632848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.692090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.732526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.789657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.867201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.941384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.008447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.048139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.546176] LNet: 62422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.553431] LNetError: 62422:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.563121] LNet: Removed LNI 192.168.202.42@tcp [ 1307.578156] LNet: Removed LNI 192.168.202.42@tcp1 [ 1308.018605] Key type .llcrypt unregistered [ 1308.022901] Key type ._llcrypt unregistered [ 1328.138497] Key type ._llcrypt registered [ 1328.140715] Key type .llcrypt registered [ 1329.826031] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:28:25 (1782242905) [ 1331.467460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1332.917322] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:28:29 (1782242909) [ 1333.671298] Key type .llcrypt unregistered [ 1333.681709] Key type ._llcrypt unregistered [ 1340.559790] Key type ._llcrypt registered [ 1340.564617] Key type .llcrypt registered [ 1340.617569] LNet: 63754:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1340.617569] mask usage: [+|-] ... [ 1340.882151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.909445] alg: No test for adler32 (adler32-zlib) [ 1341.816145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1341.848531] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1349.609614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1350.499148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1350.524599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1350.557569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1350.580890] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1350.828776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.868484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1350.895606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.930768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.972827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.013499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.063381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.075575] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1351.083343] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1351.122401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.153968] LNet: 63968:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1351.186936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.251488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.291575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.341158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.411685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.460426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.808841] LNet: 64007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.816872] LNetError: 64007:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.834403] LNet: Removed LNI 192.168.202.42@tcp [ 1352.150520] Key type .llcrypt unregistered [ 1352.165544] Key type ._llcrypt unregistered [ 1372.944609] Key type ._llcrypt registered [ 1372.946797] Key type .llcrypt registered [ 1372.992846] LNet: 64458:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1372.992846] mask usage: [+|-] ... [ 1373.192982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1373.227626] alg: No test for adler32 (adler32-zlib) [ 1374.204517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1374.239697] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1382.712704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1391.757341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1391.807106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1391.855462] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1393.651603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1393.677864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1393.715959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1393.755861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1393.794666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e network_timeout -r 1 [ 1393.836545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e network_timeout -r 1 [ 1393.887785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e network_timeout -r 1 [ 1393.948877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e network_timeout -r 1 [ 1394.008989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1394.040030] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1394.269072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1394.305692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1394.342904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1394.392572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1394.421567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1394.456757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1394.486365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1394.513377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1394.525541] LNet: 64889:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1394.540422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1394.565251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1394.592475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1394.614305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1394.633607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1394.664635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.703280] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1399.849154] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1400.006327] LNet: 64963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.013516] LNetError: 64963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.055194] LNet: Removed LNI 192.168.202.42@tcp [ 1401.057165] LNetError: 64963:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.079078] LNet: Removed LNI 192.168.202.42@tcp1 [ 1401.554583] Key type .llcrypt unregistered [ 1401.564913] Key type ._llcrypt unregistered [ 1418.474585] Key type ._llcrypt registered [ 1418.476904] Key type .llcrypt registered [ 1420.039749] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:29:55 (1782242995) [ 1420.283875] LNet: 65765:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1420.283875] mask usage: [+|-] ... [ 1420.456733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.466945] alg: No test for adler32 (adler32-zlib) [ 1421.343089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.413408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.437503] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1422.119779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1422.149760] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1422.933926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1423.001943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1423.047450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1423.074676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1423.107183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1423.139636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1423.168644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1423.192893] LNetError: 66064:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1423.456045] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1423.458057] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1425.503490] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1426.707475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1429.604643] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1430.204720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1430.235235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.293907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.314643] LNet: 66171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.326343] LNetError: 66171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.391148] LNet: Removed LNI 192.168.202.42@tcp [ 1431.394691] LNetError: 66171:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.415780] LNet: Removed LNI 192.168.202.42@tcp1 [ 1431.454640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.531439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.554890] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1432.350272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.380977] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1433.199371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1433.226412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1433.269839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1433.304169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1433.334758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1433.367317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1433.397299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1433.424036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1433.445978] LNetError: 66444:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1433.567890] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1435.924616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1439.711427] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1439.722024] LNet: Skipped 1 previous similar message [ 1445.069996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1445.122617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1445.165327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.664918] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:30:28 (1782243028) [ 1452.778140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.822099] LNet: 67154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.840905] LNetError: 67154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.855630] LNet: Removed LNI 192.168.202.42@tcp [ 1453.876631] LNet: Removed LNI 192.168.202.42@tcp1 [ 1453.915101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.045487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1454.106942] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1455.287940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1455.320187] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1456.392165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1456.474267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1456.510940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1456.538836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1456.577355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1456.606436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1456.653808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1456.758905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1457.055382] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.42@tcp [ 1457.055489] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1457.083140] LNet: Skipped 1 previous similar message [ 1471.620618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.901560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1472.064128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1472.176658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 500 [ 1472.288466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1479.084409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1479.163606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.181993] LNet: 67647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.190520] LNetError: 67647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.201292] LNetError: 67647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1479.204220] LNet: Removed LNI 192.168.202.42@tcp [ 1479.234821] LNet: Removed LNI 192.168.202.42@tcp1 [ 1479.252686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.289879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.312315] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1479.853892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1479.871196] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1480.640353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1480.669811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1480.723559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1480.746111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1480.774956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1480.790288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1480.809538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1480.856869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.42@tcp --health 0 [ 1481.311393] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1481.316753] LNet: Skipped 3 previous similar messages [ 1482.148636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.552783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.594387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1492.631427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1499.402310] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:31:15 (1782243075) [ 1500.100732] LNet: 68693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.106312] LNetError: 68693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.111205] LNetError: 68693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1500.115814] LNet: Removed LNI 192.168.202.42@tcp [ 1500.124625] LNet: Removed LNI 192.168.202.42@tcp1 [ 1500.684699] Key type .llcrypt unregistered [ 1500.687840] Key type ._llcrypt unregistered [ 1507.218227] Key type ._llcrypt registered [ 1507.220744] Key type .llcrypt registered [ 1507.255866] LNet: 68916:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1507.255866] mask usage: [+|-] ... [ 1507.490888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.521402] alg: No test for adler32 (adler32-zlib) [ 1508.502524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1508.525556] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1518.008983] Lustre: DEBUG MARKER: Initial discovery [ 1518.121825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1520.752622] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1520.879554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1520.932467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1520.973787] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1521.005861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1521.061483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -l 3 [ 1521.115405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.142@tcp -r 1 -m GET -e local_error [ 1521.169470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.142@tcp -r 1 -m PUT -l 6 [ 1521.827372] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1522.639654] Lustre: DEBUG MARKER: Force oleg242-client.virtnet to discover 192.168.202.142@tcp (in background) [ 1522.839108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.142@tcp [ 1523.936851] Lustre: DEBUG MARKER: Force oleg242-server to discover 192.168.202.42@tcp [ 1525.923535] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1526.708732] Lustre: DEBUG MARKER: Wait for 69454 [ 1528.020258] Lustre: DEBUG MARKER: Finished wait on 69454 [ 1528.200323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1528.289113] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1528.296376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1528.518867] LNet: 69905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.526904] LNetError: 69905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.578650] LNet: Removed LNI 192.168.202.42@tcp [ 1530.052306] Key type .llcrypt unregistered [ 1530.055173] Key type ._llcrypt unregistered [ 1547.664721] Key type ._llcrypt registered [ 1547.668706] Key type .llcrypt registered [ 1548.756825] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:32:04 (1782243124) [ 1549.378323] Key type .llcrypt unregistered [ 1549.380283] Key type ._llcrypt unregistered [ 1555.357240] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1555.557667] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1555.563299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1555.570056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1558.223908] Key type ._llcrypt registered [ 1558.226347] Key type .llcrypt registered [ 1558.276386] LNet: 70944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1558.276386] mask usage: [+|-] ... [ 1558.442408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1558.463204] alg: No test for adler32 (adler32-zlib) [ 1559.343965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.405664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.425960] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1560.104277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1560.130322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1567.635920] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:32:23 (1782243143) [ 1567.772491] LNet: 71973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.778559] LNetError: 71973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1567.786892] LNet: Removed LNI 10.1.2.3@tcp [ 1567.797266] LNet: Removed LNI 192.168.202.42@tcp [ 1568.459820] Key type .llcrypt unregistered [ 1568.464571] Key type ._llcrypt unregistered [ 1575.534431] Key type ._llcrypt registered [ 1575.537601] Key type .llcrypt registered [ 1575.582881] LNet: 72227:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1575.582881] mask usage: [+|-] ... [ 1575.758594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.774594] alg: No test for adler32 (adler32-zlib) [ 1576.776386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.855693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.906807] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1577.726643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1577.748846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1578.746341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.42@tcp [ 1579.807769] LNetError: Unexpected error -22 connecting to 192.168.202.42@tcp at host 192.168.202.42:988 [ 1584.871901] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:32:40 (1782243160) [ 1585.252489] LNet: 73110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.260852] LNetError: 73110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1585.268577] LNet: Removed LNI 10.1.2.3@tcp [ 1586.281651] LNet: Removed LNI 192.168.202.42@tcp [ 1586.901441] Key type .llcrypt unregistered [ 1586.903514] Key type ._llcrypt unregistered [ 1593.085695] Key type ._llcrypt registered [ 1593.088918] Key type .llcrypt registered [ 1593.136732] LNet: 73341:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1593.136732] mask usage: [+|-] ... [ 1593.309964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.321709] alg: No test for adler32 (adler32-zlib) [ 1594.270849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.334964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1594.366971] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1594.982272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1594.996904] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 1595.882887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.42@tcp1 --nid 192.168.202.42@tcp2 [ 1600.230285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp2 [ 1600.247962] LNetError: 73945:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.42@tcp1 to 192.168.202.42@tcp2 - Route table may be misconfigured [ 1600.286912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp2 192.168.202.42@tcp1 [ 1607.744570] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:33:03 (1782243183) [ 1607.843370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.863546] LNet: 74512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.883842] LNetError: 74512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.896681] LNet: Removed LNI 192.168.202.42@tcp1 [ 1608.959814] LNet: Removed LNI 192.168.202.42@tcp2 [ 1609.009410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.119866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.154117] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1610.019584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.038509] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1610.804460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 1610.848969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.42@tcp [ 1610.901742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp1 192.168.202.42@tcp1 [ 1610.945587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp -e network_timeout [ 1610.991730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -e network_timeout [ 1611.044740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -e network_timeout [ 1611.114860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -e network_timeout [ 1611.164787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1611.185814] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1611.219699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1611.295558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1611.389574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1612.127179] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1617.462730] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:33:13 (1782243193) [ 1617.752308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.765770] LNet: 75381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.772415] LNetError: 75381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.774471] LNetError: 75381:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1617.779153] LNet: Removed LNI 192.168.202.42@tcp [ 1618.798933] LNet: Removed LNI 192.168.202.42@tcp1 [ 1618.855261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.971850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1619.276877] LNet: 75418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.636769] Key type .llcrypt unregistered [ 1619.639713] Key type ._llcrypt unregistered [ 1632.008893] Key type ._llcrypt registered [ 1632.010862] Key type .llcrypt registered [ 1633.282600] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:33:29 (1782243209) [ 1633.766846] LNet: 76190:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1633.766846] mask usage: [+|-] ... [ 1633.901878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1633.915618] alg: No test for adler32 (adler32-zlib) [ 1634.921334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.366246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1638.250476] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:33:34 (1782243214) [ 1638.602934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.641057] LNet: 76517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.667909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.767282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1638.793471] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1639.376208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1639.390850] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1640.071249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1640.112733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp1 [ 1640.144050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp1 [ 1645.520840] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:33:41 (1782243221) [ 1647.161236] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg242-server [ 1649.388438] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:33:45 (1782243225) [ 1651.540610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg242-server [ 1653.485834] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:33:49 (1782243229) [ 1655.590839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg242-server [ 1657.393279] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:33:53 (1782243233) [ 1658.882814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg242-server [ 1660.566978] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:33:56 (1782243236) [ 1663.219641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg242-server [ 1664.692576] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:34:00 (1782243240) [ 1666.997666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg242-server [ 1668.901974] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:34:04 (1782243244) [ 1671.431244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg242-server [ 1673.604165] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:34:09 (1782243249) [ 1676.046707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg242-server [ 1678.278615] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:34:13 (1782243253) [ 1680.621568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg242-server [ 1682.872672] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:34:18 (1782243258) [ 1683.052904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.088057] LNet: 80394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.106563] LNetError: 80394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.113100] LNet: Removed LNI 192.168.202.42@tcp [ 1684.150416] LNet: Removed LNI 192.168.202.42@tcp1 [ 1684.225290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.328649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.383143] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1685.249595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1685.357940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1685.628492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.667186] LNet: 80543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.682546] LNetError: 80543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.703489] LNetError: 80543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1685.727465] LNet: Removed LNI 192.168.202.42@tcp [ 1685.736507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.792331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1685.820709] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1686.693759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1686.868915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.891230] LNetError: 80688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.907386] LNet: Removed LNI 192.168.202.42@tcp [ 1686.922288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.991173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.013909] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1687.834326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1687.931649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1688.175246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.206688] LNet: 80836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.210715] LNet: 80836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1689.265383] LNet: Removed LNI 192.168.202.42@tcp [ 1689.306745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.392567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1689.432857] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1690.284813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1690.548191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.591109] LNetError: 80981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.607274] LNetError: 80981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1690.632901] LNet: Removed LNI 192.168.202.42@tcp [ 1690.704137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.783185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.812258] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1691.358427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1691.461712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1691.654845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.711798] LNet: Removed LNI 192.168.202.42@tcp [ 1692.765902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.852100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1692.872863] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1693.778886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1694.084204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.124632] LNet: 81273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.136996] LNet: 81273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1695.212931] LNet: Removed LNI 192.168.202.42@tcp [ 1695.297981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.385689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1695.430387] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1696.513769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1696.668087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1696.992716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.033284] LNetError: 81421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.038042] LNetError: 81421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1697.052804] LNet: Removed LNI 192.168.202.42@tcp [ 1697.092037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.180778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1697.256459] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1698.501865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1700.047413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.097279] LNet: Removed LNI 192.168.202.42@tcp [ 1701.156885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.280227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.328771] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1702.272739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1702.365884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1703.780096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.800714] LNet: 81732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.806410] LNet: 81732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1704.870690] LNet: Removed LNI 192.168.202.42@tcp [ 1704.888699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.004341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1705.041602] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1706.130240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1706.389192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.430701] LNetError: 81877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.440576] LNetError: 81877:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1706.466821] LNet: Removed LNI 192.168.202.42@tcp [ 1706.496977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.554639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.581407] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1707.034291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1707.077607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1707.313604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.393759] LNet: Removed LNI 192.168.202.42@tcp [ 1708.429919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.496278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1708.518738] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1709.382322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1709.648354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.693615] LNet: Removed LNI 192.168.202.42@tcp [ 1710.720902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.780937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.801547] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1711.400583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1711.536814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.563583] LNet: Removed LNI 192.168.202.42@tcp [ 1711.584158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.659387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.707229] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1712.449816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1719.598424] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:34:55 (1782243295) [ 1719.807693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.844491] LNet: 83007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.848527] LNet: 83007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1719.863574] LNet: Removed LNI 192.168.202.42@tcp [ 1719.887725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.981511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1720.013153] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1721.376501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.424914] LNet: Removed LNI 192.168.202.42@tcp231 [ 1721.452545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.533558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1721.564920] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1721.586201] LNet: 83153:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1722.040544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1722.071574] LNet: Removed LNI 192.168.202.42@tcp231 [ 1722.083706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1722.113907] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1723.193358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.230135] LNetError: 83298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.241405] LNetError: 83298:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1723.282748] LNet: Removed LNI 192.168.202.42@tcp231 [ 1723.312234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.461670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1723.516130] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1723.529074] LNet: 83306:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.973821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1724.007757] LNet: Removed LNI 192.168.202.42@tcp231 [ 1724.036965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1724.069890] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/0] [ 1724.996770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.064901] LNet: Removed LNI 192.168.202.42@tcp231 [ 1725.103265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.202783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1725.237474] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1725.242498] LNet: 83457:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1725.844327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1725.880357] LNet: Removed LNI 192.168.202.42@tcp231 [ 1725.902817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1725.928978] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/60] [ 1726.802365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.818907] LNet: Removed LNI 192.168.202.42@tcp231 [ 1726.833735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.892471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1726.917285] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1727.237310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1727.286037] LNet: Removed LNI 192.168.202.42@tcp231 [ 1727.308291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1727.346679] LNet: Added LNI 192.168.202.42@tcp231 [8/256/0/180] [ 1734.441837] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:35:10 (1782243310) [ 1734.613365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.668295] LNet: Removed LNI 192.168.202.42@tcp231 [ 1734.695796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.917435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1734.959830] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1735.750991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1735.783589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1735.810537] LNet: Removed LNI 192.168.202.42@tcp232 [ 1735.829248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1735.848513] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1736.502655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1736.551986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1736.589191] LNet: Removed LNI 192.168.202.42@tcp232 [ 1736.610484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1736.639548] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1737.376968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1737.419891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1737.452836] LNet: Removed LNI 192.168.202.42@tcp232 [ 1737.481450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1737.505349] LNet: Added LNI 192.168.202.42@tcp232 [8/256/0/180] [ 1738.274743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1738.308474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1738.334838] LNet: Removed LNI 192.168.202.42@tcp232 [ 1738.348474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1745.660930] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:35:21 (1782243321) [ 1747.068615] Key type .llcrypt unregistered [ 1747.070103] Key type ._llcrypt unregistered [ 1753.461321] Key type ._llcrypt registered [ 1753.464576] Key type .llcrypt registered [ 1753.538924] LNet: 85676:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1753.538924] mask usage: [+|-] ... [ 1753.833743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1753.852666] alg: No test for adler32 (adler32-zlib) [ 1754.712947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1754.730634] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1762.953488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1771.359879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.415343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1771.475872] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1773.047758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1773.076305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1773.116762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1773.163365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1773.219798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_interrupt -r 2 [ 1773.298661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_interrupt -r 2 [ 1773.373586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_interrupt -r 2 [ 1773.471324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_interrupt -r 2 [ 1773.548291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1773.611888] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1773.619189] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1774.005583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1774.069308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1774.124409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1774.181889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1774.236971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.284231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1774.322367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1774.359430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1774.372438] LNet: 86109:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1774.393279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1774.420101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1774.446418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1774.468885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1774.493636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1774.548489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1775.180178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1775.216254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1775.254899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1775.293842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1775.337204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_dropped -r 2 [ 1775.402308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_dropped -r 2 [ 1775.441533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_dropped -r 2 [ 1775.485388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_dropped -r 2 [ 1775.538670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1775.567582] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1775.872733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1775.906634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1775.930187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1775.957878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1775.989737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1776.023247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1776.059473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1776.107455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1776.133479] LNet: 86229:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.163159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1776.195664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1776.227820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1776.250344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1776.301543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1776.356942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1776.971649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1777.008835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1777.027418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1777.072587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1777.131561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_aborted -r 2 [ 1777.194787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_aborted -r 2 [ 1777.241485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_aborted -r 2 [ 1777.309511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_aborted -r 2 [ 1777.382466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1777.415511] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1777.748530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1777.797719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1777.837580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1777.909306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1777.979297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1778.035497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1778.090729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1778.155195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1778.187160] LNet: 86349:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1778.205438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1778.230428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1778.248803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1778.275483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1778.299417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1778.324365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1778.855339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1778.898651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1778.951532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1778.982413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1779.009474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_no_route -r 2 [ 1779.042463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_no_route -r 2 [ 1779.075963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_no_route -r 2 [ 1779.132449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_no_route -r 2 [ 1779.193647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1779.494456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1779.530371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1779.574781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1779.617310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1779.652917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1779.724639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1779.778407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1779.818411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1779.861048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1779.906437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1779.987464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1780.025385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1780.109370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1780.167716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1780.919055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1780.952954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1780.982913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1781.018190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1781.047946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e local_timeout -r 2 [ 1781.072953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e local_timeout -r 2 [ 1781.109097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e local_timeout -r 2 [ 1781.138918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e local_timeout -r 2 [ 1781.158281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1781.174514] LNet: There was an unexpected network error while writing to 192.168.202.142: rc = -22 [ 1781.177240] LNet: Skipped 1 previous similar message [ 1781.444985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1781.506980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1781.549607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1781.600373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1781.644304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1781.698973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1781.743963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1781.798994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1781.837921] LNet: 86590:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.846232] LNet: 86590:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1781.856890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1781.890062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1781.938684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1781.981273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1782.028941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1782.057555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1782.758495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1782.852832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1782.907780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1782.947657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1783.012456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp -m GET -e remote_dropped -r 2 [ 1783.059752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.142@tcp1 -m GET -e remote_dropped -r 2 [ 1783.114266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp -m GET -e remote_dropped -r 2 [ 1783.182923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.42@tcp1 -d 192.168.202.42@tcp1 -m GET -e remote_dropped -r 2 [ 1783.229276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.42@tcp 192.168.202.142@tcp [ 1783.254781] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 1783.451756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1783.472974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1783.511048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1783.538749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1783.574441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1783.600525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1783.638454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1783.663872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1783.707059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1783.738262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1783.765867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1783.798974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1783.838981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1783.898489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1790.475257] LNet: 86785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.483226] LNetError: 86785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.488526] LNet: Removed LNI 192.168.202.42@tcp [ 1790.499581] LNet: Removed LNI 192.168.202.42@tcp1 [ 1790.868427] Key type .llcrypt unregistered [ 1790.871598] Key type ._llcrypt unregistered [ 1809.038902] Key type ._llcrypt registered [ 1809.046889] Key type .llcrypt registered [ 1810.249614] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:36:26 (1782243386) [ 1810.707928] LNet: 87590:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1810.707928] mask usage: [+|-] ... [ 1810.942512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1810.963971] alg: No test for adler32 (adler32-zlib) [ 1811.925633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.979310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.015933] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1813.004460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1813.037669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.42@tcp [ 1813.062164] LNetError: 87763:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1813.174707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.42@tcp [ 1813.321644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.637836] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:36:35 (1782243395) [ 1819.827711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.868255] LNet: 88350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.881023] LNetError: 88350:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.927207] LNet: Removed LNI 192.168.202.42@tcp [ 1819.960658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.047841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.089204] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1820.732780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1820.759363] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 1821.679122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.42@tcp [ 1821.695676] LNetError: 88618:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.42@tcp to recovery [ 1822.047589] LNet: 1 local NIs in recovery (showing 1): 192.168.202.42@tcp [ 1822.048983] LNet: There was an unexpected network error while writing to 192.168.202.42: rc = -22 [ 1822.942941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.42@tcp [ 1831.515685] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:36:47 (1782243407) [ 1833.363985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg242-server [ 1835.093702] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:36:51 (1782243411) [ 1837.217387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg242-server [ 1838.819335] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:36:54 (1782243414) [ 1840.898847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg242-server [ 1842.750177] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:36:58 (1782243418) [ 1842.855990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.902962] LNet: 90243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.916794] LNetError: 90243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.935338] LNet: Removed LNI 192.168.202.42@tcp [ 1843.941239] LNetError: 90243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.963061] LNet: Removed LNI 192.168.202.42@tcp1 [ 1844.031525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.154650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.192082] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1845.240283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1845.307356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1845.340107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1845.396529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1845.443963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1845.481873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1851.796769] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:37:07 (1782243427) [ 1851.918604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.947158] LNet: 90971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.954133] LNetError: 90971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.976729] LNet: Removed LNI 192.168.202.42@tcp [ 1852.018448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.061883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.080187] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1859.224893] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:37:14 (1782243434) [ 1859.311728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.331141] LNet: 91673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.336050] LNetError: 91673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.371320] LNet: Removed LNI 192.168.202.42@tcp [ 1859.415943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.499760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1859.521969] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1867.596320] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:37:23 (1782243443) [ 1869.256807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1870.764757] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:37:26 (1782243446) [ 1871.160885] LNet: 92715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.165326] LNetError: 92715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.177257] LNet: Removed LNI 192.168.202.42@tcp [ 1871.801910] Key type .llcrypt unregistered [ 1871.810422] Key type ._llcrypt unregistered [ 1879.186843] Key type ._llcrypt registered [ 1879.193233] Key type .llcrypt registered [ 1879.226407] LNet: 92944:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1879.226407] mask usage: [+|-] ... [ 1879.490474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1879.527173] alg: No test for adler32 (adler32-zlib) [ 1880.552691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1880.601161] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1888.984728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 1893.923197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.142@tcp [ 1894.334714] LNet: 93133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1894.341793] LNetError: 93133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1894.358442] LNet: Removed LNI 192.168.202.42@tcp [ 1894.812414] Key type .llcrypt unregistered [ 1894.814212] Key type ._llcrypt unregistered [ 1906.878605] Key type ._llcrypt registered [ 1906.885681] Key type .llcrypt registered [ 1908.192834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1910.109280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1912.392744] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:38:07 (1782243487) [ 1914.412462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1916.529958] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:38:12 (1782243492) [ 1918.616989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg242-server [ 1920.267883] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:38:16 (1782243496) [ 1922.293545] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg242-server [ 1924.007628] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:38:19 (1782243499) [ 1925.006256] LNet: 95183:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1925.006256] mask usage: [+|-] ... [ 1925.714645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.783273] alg: No test for adler32 (adler32-zlib) [ 1926.849158] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1927.068574] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1928.799148] Key type lgssc registered [ 1929.865675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.624929] Key type lgssc unregistered [ 1951.953707] LNet: 96310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.964709] LNetError: 96310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.986766] LNet: Removed LNI 192.168.202.42@tcp [ 1952.596525] Key type .llcrypt unregistered [ 1952.599411] Key type ._llcrypt unregistered [ 1967.552367] Key type ._llcrypt registered [ 1967.554197] Key type .llcrypt registered [ 1968.724144] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:39:04 (1782243544) [ 1969.085731] LNet: 97108:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1969.085731] mask usage: [+|-] ... [ 1969.240723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.253793] alg: No test for adler32 (adler32-zlib) [ 1970.196182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.293882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.362898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.453743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1970.557740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1977.015357] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:39:13 (1782243553) [ 1977.100961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.138076] LNet: 97719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.165452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.265710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.281062] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1978.100748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.118490] LNet: 97870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.123949] LNetError: 97870:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.137811] LNet: Removed LNI 192.168.202.42@tcp [ 1978.154617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.192221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1978.211341] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1978.475985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.510807] LNet: Removed LNI 192.168.202.42@tcp [ 1978.519991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.563485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1978.579277] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1985.329406] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:39:21 (1782243561) [ 1985.465049] LNet: 98479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.470777] LNet: 98479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1985.488213] LNetError: 98479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.496165] LNetError: 98479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1985.535970] LNet: Removed LNI 192.168.202.42@tcp [ 1986.184626] Key type .llcrypt unregistered [ 1986.188885] Key type ._llcrypt unregistered [ 1994.385218] Key type ._llcrypt registered [ 1994.387403] Key type .llcrypt registered [ 1994.423169] LNet: 98719:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1994.423169] mask usage: [+|-] ... [ 1994.619539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1994.633867] alg: No test for adler32 (adler32-zlib) [ 1995.562662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.641766] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 1995.645453] LNet: *** cfs_fail_loc=e003, val=100*** [ 1996.339635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 1996.360087] LNet: *** cfs_fail_loc=e003, val=100*** [ 1996.364340] LNet: Skipped 2 previous similar messages [ 1996.374279] Ping reply received with mapping: [ 1996.377320] found metadata reply NI[192.168.202.42@tcp] [ 1996.381546] 2a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1996.388667] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1996.390494] 5a 5a 5a 5a 5a 5a 5a 5a [ 1996.514402] LNet: 98887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1996.521851] LNetError: 98887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.543548] LNet: Removed LNI 192.168.202.42@tcp [ 1998.352416] Key type .llcrypt unregistered [ 1998.356660] Key type ._llcrypt unregistered [ 2010.429796] Key type ._llcrypt registered [ 2010.432153] Key type .llcrypt registered [ 2011.640826] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:39:47 (1782243587) [ 2012.285394] Key type .llcrypt unregistered [ 2012.288533] Key type ._llcrypt unregistered [ 2019.503972] Key type ._llcrypt registered [ 2019.510651] Key type .llcrypt registered [ 2019.560600] LNet: 99882:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2019.560600] mask usage: [+|-] ... [ 2019.730538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2019.740365] alg: No test for adler32 (adler32-zlib) [ 2020.779962] LNetError: 99932:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2021.257658] Key type .llcrypt unregistered [ 2021.261793] Key type ._llcrypt unregistered [ 2027.251173] Key type ._llcrypt registered [ 2027.254396] Key type .llcrypt registered [ 2028.584460] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:40:04 (1782243604) [ 2029.289411] Key type .llcrypt unregistered [ 2029.290618] Key type ._llcrypt unregistered [ 2036.146272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2036.165982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2039.141823] Key type ._llcrypt registered [ 2039.151078] Key type .llcrypt registered [ 2039.251897] LNet: 100800:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2039.251897] mask usage: [+|-] ... [ 2039.597054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.619175] alg: No test for adler32 (adler32-zlib) [ 2040.769376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.880642] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2054.474291] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:40:30 (1782243630) [ 2054.577908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.620617] LNet: 101709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.631436] LNetError: 101709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2054.670245] LNet: Removed LNI 10.1.2.3@tcp [ 2054.727098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.119342] Key type .llcrypt unregistered [ 2056.133151] Key type ._llcrypt unregistered [ 2064.199933] Key type ._llcrypt registered [ 2064.202567] Key type .llcrypt registered [ 2064.240600] LNet: 101945:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2064.240600] mask usage: [+|-] ... [ 2064.379643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2064.399317] alg: No test for adler32 (adler32-zlib) [ 2065.478608] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2065.570894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2065.793635] LNet: 102002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.800980] LNetError: 102002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2065.826707] LNet: Removed LNI 192.168.202.42@tcp [ 2066.867108] Key type .llcrypt unregistered [ 2066.869400] Key type ._llcrypt unregistered [ 2078.620902] Key type ._llcrypt registered [ 2078.623413] Key type .llcrypt registered [ 2080.541864] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:40:55 (1782243655) [ 2081.384204] Key type .llcrypt unregistered [ 2081.386138] Key type ._llcrypt unregistered [ 2089.467848] Key type ._llcrypt registered [ 2089.471411] Key type .llcrypt registered [ 2089.521940] LNet: 102987:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2089.521940] mask usage: [+|-] ... [ 2089.688917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2089.709752] alg: No test for adler32 (adler32-zlib) [ 2097.674045] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:41:13 (1782243673) [ 2097.824682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.883630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.003730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2098.050670] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2098.602233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2098.639600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2098.671519] LNet: Removed LNI 192.168.202.42@tcp [ 2098.881182] LNet: 103746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.153504] Key type .llcrypt unregistered [ 2099.157688] Key type ._llcrypt unregistered [ 2111.636235] Key type ._llcrypt registered [ 2111.638381] Key type .llcrypt registered [ 2113.244549] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:41:28 (1782243688) [ 2113.550016] LNet: 104510:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2113.550016] mask usage: [+|-] ... [ 2113.745354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2113.762682] alg: No test for adler32 (adler32-zlib) [ 2114.738217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.789922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.822964] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2115.626166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2115.719550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.42@tcp [ 2115.744915] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2115.750329] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2115.752691] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2115, 0, 0 [ 2115.755954] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2115.757941] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2115.760912] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2115.763622] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2115, 0, 0 [ 2115.767809] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2115.773930] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2115.778612] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2115.784980] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2115, 0, 2165 [ 2115.792942] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2115.795472] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2115.797584] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2115.803436] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2115, 0, 2165 [ 2115.809403] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2122.258885] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:41:38 (1782243698) [ 2122.839533] LNet: 105286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.846940] LNetError: 105286:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.867606] LNet: Removed LNI 192.168.202.42@tcp [ 2123.686440] Key type .llcrypt unregistered [ 2123.690242] Key type ._llcrypt unregistered [ 2130.615700] Key type ._llcrypt registered [ 2130.619340] Key type .llcrypt registered [ 2130.696847] LNet: 105509:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2130.696847] mask usage: [+|-] ... [ 2130.937744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2130.965363] alg: No test for adler32 (adler32-zlib) [ 2131.931309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2131.965418] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2139.815655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2148.691890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.719762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2148.756923] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 2149.852151] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2149.933104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2149.936503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2149.952393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2152.419306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2152.449617] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2165.194900] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:42:21 (1782243741) [ 2165.327250] LNet: 106782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.340994] LNetError: 106782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.357064] LNet: Removed LNI 192.168.202.42@tcp [ 2166.367295] LNet: Removed LNI 192.168.202.42@tcp1 [ 2166.369841] LNetError: 106782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2166.378490] LNetError: 106782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2166.403468] LNet: Removed LNI 10.1.2.3@tcp99 [ 2167.298535] Key type .llcrypt unregistered [ 2167.302775] Key type ._llcrypt unregistered [ 2174.424696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2174.432081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2177.139784] Key type ._llcrypt registered [ 2177.143668] Key type .llcrypt registered [ 2177.165754] LNet: 107055:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2177.165754] mask usage: [+|-] ... [ 2177.325553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2177.344296] alg: No test for adler32 (adler32-zlib) [ 2178.291157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.382930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.413504] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2179.262216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2179.280549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2181.759827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2181.824234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2181.871133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2181.897588] LNetError: 107533:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2181.932720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2181.976466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2182.371554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2182.483866] LNet: 107553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.489514] LNetError: 107553:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2182.499667] LNet: Removed LNI 10.1.2.3@tcp [ 2183.536484] LNet: Removed LNI 192.168.202.42@tcp [ 2184.251254] Key type .llcrypt unregistered [ 2184.253496] Key type ._llcrypt unregistered [ 2199.529367] Key type ._llcrypt registered [ 2199.533599] Key type .llcrypt registered [ 2201.675449] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:42:56 (1782243776) [ 2202.099830] LNet: 108350:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2202.099830] mask usage: [+|-] ... [ 2202.362803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2202.383533] alg: No test for adler32 (adler32-zlib) [ 2203.404107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2203.486154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2203.526978] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2204.693863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.42@tcp [ 2204.803617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg242-client@tcp [ 2212.768575] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:43:08 (1782243788) [ 2212.950820] LNet: 109089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.955529] LNetError: 109089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.965523] LNet: Removed LNI 192.168.202.42@tcp [ 2213.830591] Key type .llcrypt unregistered [ 2213.833309] Key type ._llcrypt unregistered [ 2221.828184] Key type ._llcrypt registered [ 2221.835500] Key type .llcrypt registered [ 2222.002851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2222.029223] alg: No test for adler32 (adler32-zlib) [ 2222.899297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2222.915601] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2222.942472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2230.655900] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:43:26 (1782243806) [ 2232.375343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2234.204771] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:43:30 (1782243810) [ 2235.832751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2237.714252] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:43:33 (1782243813) [ 2237.822209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.851336] LNet: 110502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.858811] LNetError: 110502:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.873686] LNet: Removed LNI 192.168.202.42@tcp [ 2237.901853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.011341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.059548] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2238.934341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2238.991464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2245.920798] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:43:41 (1782243821) [ 2246.219257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.239868] LNet: 111218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.245557] LNetError: 111218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.265034] LNet: Removed LNI 192.168.202.42@tcp [ 2246.298682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.343386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2246.374515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2246.975376] Key type .llcrypt unregistered [ 2246.977887] Key type ._llcrypt unregistered [ 2259.157097] Key type ._llcrypt registered [ 2259.159709] Key type .llcrypt registered [ 2260.497790] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:43:56 (1782243836) [ 2260.952774] LNet: 112025:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2260.952774] mask usage: [+|-] ... [ 2261.140907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.151193] alg: No test for adler32 (adler32-zlib) [ 2262.122289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.214340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.278963] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2263.402680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2263.477749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2271.331711] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:44:07 (1782243847) [ 2271.560126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.605147] LNet: 112774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.611126] LNetError: 112774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2272.679703] LNet: Removed LNI 192.168.202.42@tcp [ 2272.717081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.806495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2272.841746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2279.333290] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:44:15 (1782243855) [ 2279.827313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.869283] LNet: 113354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.890600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.974212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2280.356069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2280.395901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2280.461830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2281.089953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.147096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.262399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.299777] LNet: 113388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.340243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.440995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2281.757747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2281.806884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.888230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2282.529169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2282.587875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2289.122838] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:44:25 (1782243865) [ 2289.306495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.337170] LNet: 113983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.375511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.510299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.547840] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2290.430901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2290.450917] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2291.325204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2291.390895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2291.645437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.670886] LNetError: 114268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.674824] LNet: Removed LNI 192.168.202.42@tcp [ 2291.684639] LNet: Removed LNI 192.168.202.42@tcp2 [ 2291.709765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.775763] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2291.784895] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2291.793426] LNet: 114274:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2292.481730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2292.684379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.707724] LNetError: 114311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.713716] LNetError: 114311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2292.719766] LNet: Removed LNI 192.168.202.42@tcp [ 2292.730117] LNet: Removed LNI 192.168.202.42@tcp2 [ 2292.745706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.797964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.825191] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2293.771430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2293.793623] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2295.227880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2295.325590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2295.450086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2295.732572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.760916] LNet: 114596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.766810] LNet: 114596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2295.777645] LNetError: 114596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.787429] LNetError: 114596:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2295.804964] LNet: Removed LNI 192.168.202.42@tcp [ 2296.879595] LNet: Removed LNI 192.168.202.42@tcp2 [ 2296.955154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.038111] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2297.043872] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2297.052503] LNet: 114602:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.870664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2298.154318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.182589] LNet: Removed LNI 192.168.202.42@tcp [ 2299.239328] LNet: Removed LNI 192.168.202.42@tcp2 [ 2299.352611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.580770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.641684] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2300.670558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2300.713791] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2301.715585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2301.783694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2301.995333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2302.018348] LNetError: 114919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2302.022550] LNetError: 114919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2302.030920] LNet: Removed LNI 192.168.202.42@tcp [ 2302.038237] LNet: Removed LNI 192.168.202.42@tcp2 [ 2302.052983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.120561] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2302.130800] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2302.137606] LNet: 114925:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2302.656859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2302.931699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.968101] LNet: Removed LNI 192.168.202.42@tcp [ 2303.989073] LNet: Removed LNI 192.168.202.42@tcp2 [ 2304.050655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.142823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.246914] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2305.310262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2305.350547] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2306.446190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2306.591903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2306.678992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2306.918798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.932684] LNet: 115246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2306.937656] LNet: 115246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2307.999697] LNet: Removed LNI 192.168.202.42@tcp [ 2309.033486] LNet: Removed LNI 192.168.202.42@tcp2 [ 2309.114878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.290837] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2309.308070] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2309.336486] LNet: 115253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2310.201352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2310.410852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.435290] LNetError: 115290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.440478] LNetError: 115290:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2310.447114] LNet: Removed LNI 192.168.202.42@tcp [ 2311.461779] LNet: Removed LNI 192.168.202.42@tcp2 [ 2311.502882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.616301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.663850] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2312.697809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2312.728888] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2314.163780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2314.223448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2314.319552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2314.752453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.807192] LNet: Removed LNI 192.168.202.42@tcp [ 2315.816643] LNet: Removed LNI 192.168.202.42@tcp2 [ 2315.828601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.904163] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2315.914540] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2315.919726] LNet: 115593:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2316.373723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2316.779981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2317.855376] LNet: Removed LNI 192.168.202.42@tcp [ 2317.867529] LNet: Removed LNI 192.168.202.42@tcp2 [ 2317.883652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.946911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.993853] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2318.939876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2318.956103] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2320.043992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2320.072786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2320.098304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2320.155227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2320.598600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2321.631394] LNet: Removed LNI 192.168.202.42@tcp [ 2322.670712] LNet: Removed LNI 192.168.202.42@tcp2 [ 2322.706761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2322.821157] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2322.833523] LNet: Added LNI 192.168.202.42@tcp2 [8/256/0/180] [ 2323.541617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2331.208779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2333.666762] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:45:09 (1782243909) [ 2333.794990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.817608] LNet: 116706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.825128] LNet: 116706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2333.843391] LNetError: 116706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2333.854857] LNetError: 116706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2333.868280] LNet: Removed LNI 192.168.202.42@tcp [ 2334.885947] LNet: Removed LNI 192.168.202.42@tcp2 [ 2334.937223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.325587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2335.354680] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2335.965460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2335.981112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2336.631484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2336.644448] LNet: Removed LNI 10.1.2.3@tcp [ 2336.651826] LNet: Removed LNI 192.168.202.42@tcp [ 2336.663605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.702540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2336.735586] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2336.742390] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2337.203533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2337.221664] LNet: Removed LNI 192.168.202.42@tcp [ 2337.241922] LNet: Removed LNI 10.1.2.3@tcp [ 2344.652431] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:45:20 (1782243920) [ 2344.847681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.934714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.054059] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2345.243371] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2345.257722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2345.266432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2347.928696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.979657] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2349.102483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2349.130905] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2351.743920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.42@tcp [ 2351.793041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2351.847953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.42@tcp -m PUT -e local_timeout -r 1 [ 2351.947635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2352.028360] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2352.038979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2352.283351] LNetError: 117646:0:(lib-move.c:4714:lnet_parse()) 192.168.202.42@tcp, src 192.168.202.42@tcp: Dropping PUT (error -108 looking up sender) [ 2353.314108] LNet: Removed LNI 10.1.2.3@tcp [ 2354.354951] LNet: Removed LNI 192.168.202.42@tcp [ 2355.027850] Key type .llcrypt unregistered [ 2355.034494] Key type ._llcrypt unregistered [ 2369.445770] Key type ._llcrypt registered [ 2369.451355] Key type .llcrypt registered [ 2371.337114] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:45:46 (1782243946) [ 2371.585936] LNet: 118874:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2371.585936] mask usage: [+|-] ... [ 2371.751712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.782573] alg: No test for adler32 (adler32-zlib) [ 2372.699721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.172777] LNet: 118957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.945644] Key type .llcrypt unregistered [ 2373.949525] Key type ._llcrypt unregistered [ 2380.926681] Key type ._llcrypt registered [ 2380.932288] Key type .llcrypt registered [ 2381.031929] LNet: 119183:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2381.031929] mask usage: [+|-] ... [ 2381.234826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2381.252124] alg: No test for adler32 (adler32-zlib) [ 2382.126318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2382.142064] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2389.982641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2390.219856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2390.760653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2390.942219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2390.971404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2390.994374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2391.026985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2391.063209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2391.121868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2391.183777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2391.244153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2391.273693] LNet: 119404:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2391.304755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2391.373493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2391.391344] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 2391.439245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2391.490821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2391.528868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2391.561563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2392.251758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2392.587798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2392.641893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2392.697813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2392.742994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2392.788477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2392.823880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2392.849451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2392.887956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2392.912992] LNet: 119521:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2392.935556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2392.997557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2393.067811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2393.115318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2393.155984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2393.228667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2393.966121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2394.352567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2394.397725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2394.422525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2394.454922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2394.508586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2394.544513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2394.563405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2394.582810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2394.622353] LNet: 119638:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2394.651378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2394.689091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2394.719394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2394.757433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2394.791082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2394.834286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2395.669823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2395.871085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2395.896035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2395.923428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2395.976655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2396.013529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2396.053296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2396.090427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2396.144991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2396.212861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2396.257494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2396.286554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2396.328227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2396.379346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2396.418814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2396.749336] LNet: 119802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.758997] LNetError: 119802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.792075] LNet: Removed LNI 192.168.202.42@tcp [ 2397.300683] Key type .llcrypt unregistered [ 2397.302485] Key type ._llcrypt unregistered [ 2417.965269] Key type ._llcrypt registered [ 2417.967603] Key type .llcrypt registered [ 2419.449331] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:46:34 (1782243994) [ 2419.878389] LNet: 120608:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2419.878389] mask usage: [+|-] ... [ 2420.152880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.198495] alg: No test for adler32 (adler32-zlib) [ 2421.313246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2421.922702] LNet: 120689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.532475] Key type .llcrypt unregistered [ 2422.534101] Key type ._llcrypt unregistered [ 2429.689774] Key type ._llcrypt registered [ 2429.693691] Key type .llcrypt registered [ 2429.745736] LNet: 120918:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2429.745736] mask usage: [+|-] ... [ 2429.972658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2430.011942] alg: No test for adler32 (adler32-zlib) [ 2431.107036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2431.142275] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2439.475434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.142@tcp [ 2450.464653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.507072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2450.573466] LNet: Added LNI 192.168.202.42@tcp1 [8/256/0/180] [ 2451.545962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2452.203638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.142@tcp [ 2452.455240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2452.487170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2452.516521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2452.548778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2452.590060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2452.642244] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.142@tcp [ 2452.644448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.689830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2452.744751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2452.791900] LNet: 121342:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.835335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2452.898088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2452.953309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.023097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.072552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.132319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.926388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2454.172486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2454.223417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2454.271566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2454.299410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2454.341331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2454.364416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2454.399202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2454.438039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2454.462111] LNet: 121459:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2454.480615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2454.513336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2454.539623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2454.563278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2454.597134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2454.645707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2455.208772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.142@tcp [ 2455.404423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2455.433529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2455.461554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2455.488721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2455.513483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2455.537788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2455.557740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2455.583260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2455.600576] LNet: 121576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.616599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2455.638935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2455.666380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2455.690994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2455.722947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.758465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2456.338840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2456.545110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2456.577794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2456.601820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2456.630337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.656484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2456.682342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2456.708523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2456.734426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2456.766936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2456.785988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2456.808411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2456.828635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2456.860683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2456.885470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2462.487364] LNet: 121774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2462.493266] LNetError: 121774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2462.501975] LNet: Removed LNI 192.168.202.42@tcp [ 2462.530599] LNet: Removed LNI 192.168.202.42@tcp1 [ 2463.086831] Key type .llcrypt unregistered [ 2463.088563] Key type ._llcrypt unregistered [ 2483.515378] Key type ._llcrypt registered [ 2483.526633] Key type .llcrypt registered [ 2485.178455] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:47:40 (1782244060) [ 2493.073929] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:47:49 (1782244069) [ 2500.890608] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:47:56 (1782244076) [ 2509.850397] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:48:05 (1782244085) [ 2518.163481] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:48:13 (1782244093) [ 2525.648947] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:48:21 (1782244101) [ 2533.703480] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:48:29 (1782244109) [ 2542.150597] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:48:38 (1782244118) [ 2550.224146] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:48:46 (1782244126) [ 2560.050983] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:48:54 (1782244134) [ 2569.544762] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:49:04 (1782244144) [ 2578.342194] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:49:14 (1782244154) [ 2586.992812] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:49:22 (1782244162) [ 2594.805759] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:49:30 (1782244170) [ 2604.866737] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:49:40 (1782244180) [ 2613.254566] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:49:49 (1782244189) [ 2621.260961] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:49:57 (1782244197) [ 2630.231651] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:50:06 (1782244206) [ 2630.587901] LNet: 132320:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2630.587901] mask usage: [+|-] ... [ 2630.818608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2630.843956] alg: No test for adler32 (adler32-zlib) [ 2631.809682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2631.961910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2632.058280] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2633.360979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2633.376846] LNet: 132491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2633.384234] LNetError: 132491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.401869] LNet: Removed LNI 192.168.202.42@tcp [ 2633.443099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2633.575891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2633.617120] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2640.920173] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:50:16 (1782244216) [ 2649.713988] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:50:25 (1782244225) [ 2656.991504] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:50:33 (1782244233) [ 2664.565794] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:50:40 (1782244240) [ 2672.348835] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:50:48 (1782244248) [ 2681.583258] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:50:57 (1782244257) [ 2691.862328] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:51:07 (1782244267) [ 2700.190718] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:51:15 (1782244275) [ 2710.187413] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:51:25 (1782244285) [ 2720.607506] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:51:35 (1782244295) [ 2727.847542] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:51:44 (1782244304) [ 2735.309369] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:51:50 (1782244310) [ 2742.525369] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2473 sec ========== 15:51:58 (1782244318) [ 2742.949270] LNet: 139958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.958392] LNetError: 139958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2742.984739] LNet: Removed LNI 192.168.202.42@tcp [ 2743.894531] Key type .llcrypt unregistered [ 2743.898085] Key type ._llcrypt unregistered [ 2758.871183] Key type ._llcrypt registered [ 2758.876425] Key type .llcrypt registered [ 2758.943952] LNet: 140416:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2758.943952] mask usage: [+|-] ... [ 2759.612706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2759.653474] alg: No test for adler32 (adler32-zlib) [ 2760.741654] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2760.983474] LNet: Added LNI 192.168.202.42@tcp [8/256/0/180] [ 2762.687167] Key type lgssc registered [ 2764.027771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2832.008494] Lustre: Mounted lustre-client [ 2832.189356] LNet: 141905:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2832.189356] mask usage: [+|-] ... [ 2838.075256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2850.374096] Lustre: Unmounted lustre-client [ 2889.859956] Key type lgssc unregistered [ 2890.079728] LNet: 142889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2890.087506] LNetError: 142889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2890.109047] LNet: Removed LNI 192.168.202.42@tcp [ 2890.947278] Key type .llcrypt unregistered [ 2890.949478] Key type ._llcrypt unregistered