[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439176616 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003365] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.008921] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011119] LSM: Security Framework initializing [ 0.012034] Yama: becoming mindful. [ 0.013024] SELinux: Initializing. [ 0.014055] *** VALIDATE selinux *** [ 0.021663] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026341] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027179] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029132] *** VALIDATE tmpfs *** [ 0.030464] *** VALIDATE proc *** [ 0.032220] *** VALIDATE cgroup *** [ 0.033012] *** VALIDATE cgroup2 *** [ 0.034288] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035182] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037032] Spectre V2 : User space: Vulnerable [ 0.038007] Speculative Store Bypass: Vulnerable [ 0.041495] debug: unmapping init [mem 0xffffffffabc59000-0xffffffffabc60fff] [ 0.043172] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044819] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.045930] ... bit width: 48 [ 0.046012] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.049873] ... event mask: 000000070000000f [ 0.050262] rcu: Hierarchical SRCU implementation. [ 0.052504] smp: Bringing up secondary CPUs ... [ 0.053440] x86: Booting SMP configuration: [ 0.054018] .... node #0, CPUs: #1 #2 #3 [ 0.057145] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102173] node 0 deferred pages initialised in 40ms [ 0.104527] devtmpfs: initialized [ 0.105156] x86/mm: Memory block size: 128MB [ 0.107596] gcov: version magic: 0x41383552 [ 0.108651] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113336] pinctrl core: initialized pinctrl subsystem [ 0.114196] [ 0.114511] ************************************************************* [ 0.116010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.117009] ** ** [ 0.119009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120010] ** ** [ 0.122009] ** This means that this kernel is built to expose internal ** [ 0.124011] ** IOMMU data structures, which may compromise security on ** [ 0.125000] ** your system. ** [ 0.125000] ** ** [ 0.127011] ** If you see this message and you are not debugging the ** [ 0.129011] ** kernel, report this immediately to your vendor! ** [ 0.131010] ** ** [ 0.132012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134019] ************************************************************* [ 0.136434] NET: Registered protocol family 16 [ 0.137361] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.139053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.142053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.144338] cpuidle: using governor menu [ 0.146000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.148628] PCI: Using configuration type 1 for base access [ 0.151136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.159103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.160018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.162056] cryptd: max_cpu_qlen set to 1000 [ 0.164217] ACPI: Added _OSI(Module Device) [ 0.165009] ACPI: Added _OSI(Processor Device) [ 0.166012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.167013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.171405] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176229] ACPI: Interpreter enabled [ 0.177104] ACPI: PM: (supports S0 S3 S4 S5) [ 0.178009] ACPI: Using IOAPIC for interrupt routing [ 0.180094] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.183302] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.192011] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.194046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.196019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.200087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.205000] acpiphp: Slot [2] registered [ 0.206076] acpiphp: Slot [5] registered [ 0.208119] acpiphp: Slot [6] registered [ 0.209058] acpiphp: Slot [3] registered [ 0.210100] acpiphp: Slot [4] registered [ 0.211078] acpiphp: Slot [7] registered [ 0.212000] acpiphp: Slot [8] registered [ 0.212000] acpiphp: Slot [9] registered [ 0.214117] acpiphp: Slot [10] registered [ 0.216100] acpiphp: Slot [11] registered [ 0.217126] acpiphp: Slot [12] registered [ 0.219108] acpiphp: Slot [13] registered [ 0.220095] acpiphp: Slot [14] registered [ 0.221091] acpiphp: Slot [15] registered [ 0.222079] acpiphp: Slot [16] registered [ 0.224097] acpiphp: Slot [17] registered [ 0.225108] acpiphp: Slot [18] registered [ 0.227138] acpiphp: Slot [19] registered [ 0.229100] acpiphp: Slot [20] registered [ 0.230126] acpiphp: Slot [21] registered [ 0.232112] acpiphp: Slot [22] registered [ 0.233173] acpiphp: Slot [23] registered [ 0.234075] acpiphp: Slot [24] registered [ 0.235106] acpiphp: Slot [25] registered [ 0.237138] acpiphp: Slot [26] registered [ 0.238117] acpiphp: Slot [27] registered [ 0.240116] acpiphp: Slot [28] registered [ 0.242166] acpiphp: Slot [29] registered [ 0.243108] acpiphp: Slot [30] registered [ 0.244142] acpiphp: Slot [31] registered [ 0.246054] PCI host bridge to bus 0000:00 [ 0.247017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.249031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.250019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.251014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.253018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.255023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.259022] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263467] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.268855] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.272597] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.275020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.277027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.280030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.283533] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286768] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.289047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.291643] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.295933] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.305017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.310016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314899] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.322022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.328016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.342016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.350911] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.357015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.362017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.373016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.380543] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.382253] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.383840] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.386350] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.388357] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.392031] iommu: Default domain type: Passthrough [ 0.393308] SCSI subsystem initialized [ 0.395110] ACPI: bus type USB registered [ 0.396137] usbcore: registered new interface driver usbfs [ 0.398092] usbcore: registered new interface driver hub [ 0.400102] usbcore: registered new device driver usb [ 0.402166] pps_core: LinuxPPS API ver. 1 registered [ 0.403010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.406067] PTP clock support registered [ 0.407138] EDAC MC: Ver: 3.0.0 [ 0.409177] PCI: Using ACPI for IRQ routing [ 0.410706] NetLabel: Initializing [ 0.412007] NetLabel: domain hash size = 128 [ 0.413006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.414131] NetLabel: unlabeled traffic allowed by default [ 0.415111] vgaarb: loaded [ 0.416403] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.418009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.424000] clocksource: Switched to clocksource kvm-clock [ 0.520569] VFS: Disk quotas dquot_6.6.0 [ 0.521664] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.523330] *** VALIDATE ramfs *** [ 0.524143] *** VALIDATE hugetlbfs *** [ 0.525770] pnp: PnP ACPI init [ 0.527598] pnp: PnP ACPI: found 6 devices [ 0.541667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.543901] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.545181] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.546511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.548016] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.549463] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.551150] NET: Registered protocol family 2 [ 0.552788] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.556085] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.559487] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.562805] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.565318] TCP: Hash tables configured (established 65536 bind 65536) [ 0.567233] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.568962] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.570863] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.572853] NET: Registered protocol family 1 [ 0.574700] RPC: Registered named UNIX socket transport module. [ 0.576088] RPC: Registered udp transport module. [ 0.577153] RPC: Registered tcp transport module. [ 0.578267] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.579610] NET: Registered protocol family 44 [ 0.580698] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.581958] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.583377] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.584680] PCI: CLS 0 bytes, default 64 [ 0.585697] Unpacking initramfs... [ 1.752738] debug: unmapping init [mem 0xffff9f147cc64000-0xffff9f147ffcffff] [ 1.756507] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.758866] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.761497] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.237323] Initialise system trusted keyrings [ 2.239084] Key type blacklist registered [ 2.241096] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.248314] zbud: loaded [ 2.250611] *** VALIDATE nfs *** [ 2.251560] *** VALIDATE nfs4 *** [ 2.252774] pstore: using deflate compression [ 2.256237] Platform Keyring initialized [ 2.355500] NET: Registered protocol family 38 [ 2.357321] Key type asymmetric registered [ 2.358801] Asymmetric key parser 'x509' registered [ 2.360650] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.363664] io scheduler mq-deadline registered [ 2.365249] io scheduler kyber registered [ 2.366904] io scheduler bfq registered [ 2.369194] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.371882] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.374608] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.377158] ACPI: Power Button [PWRF] [ 2.382358] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.388183] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.417405] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.442549] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.469967] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.478361] Non-volatile memory driver v1.3 [ 2.479929] Linux agpgart interface v0.103 [ 2.525187] virtio_blk virtio1: [vda] 149488 512-byte logical blocks (76.5 MB/73.0 MiB) [ 2.528112] vda: detected capacity change from 0 to 76537856 [ 2.543249] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.545828] vdb: detected capacity change from 0 to 1073741824 [ 2.558126] libphy: Fixed MDIO Bus: probed [ 2.567555] usbcore: registered new interface driver usbserial_generic [ 2.569239] usbserial: USB Serial support registered for generic [ 2.571448] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.575308] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.576787] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.578980] mousedev: PS/2 mouse device common for all mice [ 2.581901] rtc_cmos 00:05: RTC can wake from S4 [ 2.584283] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.585443] rtc_cmos 00:05: registered as rtc0 [ 2.590732] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.590864] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.594082] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.595837] intel_pstate: CPU model not supported [ 2.600852] hid: raw HID events driver (C) Jiri Kosina [ 2.602822] usbcore: registered new interface driver usbhid [ 2.604772] usbhid: USB HID core driver [ 2.606385] drop_monitor: Initializing network drop monitor service [ 2.608649] Initializing XFRM netlink socket [ 2.610651] NET: Registered protocol family 10 [ 2.614809] Segment Routing with IPv6 [ 2.615969] NET: Registered protocol family 17 [ 2.618036] mpls_gso: MPLS GSO support [ 2.623860] RAS: Correctable Errors collector initialized. [ 2.625876] AVX version of gcm_enc/dec engaged. [ 2.627657] AES CTR mode by8 optimization enabled [ 2.700629] sched_clock: Marking stable (2700576136, 0)->(3579464626, -878888490) [ 2.704229] registered taskstats version 1 [ 2.706394] Loading compiled-in X.509 certificates [ 2.707884] zswap: loaded using pool lzo/zbud [ 2.734038] Key type big_key registered [ 2.746713] Key type encrypted registered [ 2.748601] ima: No TPM chip found, activating TPM-bypass! [ 2.750507] ima: Allocated hash algorithm: sha1 [ 2.752175] ima: No architecture policies found [ 2.753834] evm: Initialising EVM extended attributes: [ 2.755800] evm: security.selinux [ 2.757168] evm: security.ima [ 2.758306] evm: security.capability [ 2.759589] evm: HMAC attrs: 0x1 [ 2.761809] rtc_cmos 00:05: setting system clock to 2026-08-22 06:39:00 UTC (1787380740) [ 2.767840] debug: unmapping init [mem 0xffffffffacc03000-0xffffffffacdfffff] [ 2.771201] debug: unmapping init [mem 0xffffffffab982000-0xffffffffabc58fff] [ 2.781076] Write protecting the kernel read-only data: 28672k [ 2.784916] debug: unmapping init [mem 0xffffffffaa003000-0xffffffffaa1fffff] [ 2.787748] debug: unmapping init [mem 0xffffffffaa914000-0xffffffffaa9fffff] [ 2.820064] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.828115] systemd[1]: Detected virtualization kvm. [ 2.830115] systemd[1]: Detected architecture x86-64. [ 2.832071] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.862285] systemd[1]: No hostname configured. [ 2.863443] systemd[1]: Set hostname to . [ 2.865299] random: systemd: uninitialized urandom read (16 bytes read) [ 2.867689] systemd[1]: Initializing machine ID from random generator. [ 2.908593] random: ln: uninitialized urandom read (6 bytes read) [ 2.998865] random: systemd: uninitialized urandom read (16 bytes read) [ 3.001859] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.008771] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.013404] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.598090] device-mapper: uevent: version 1.0.3 [ 3.600220] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.260434] virtio_net virtio0 ens2: renamed from eth0 [ 4.323581] scsi host0: ata_piix [ 4.357237] scsi host1: ata_piix [ 4.358544] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.362291] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.611086] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.264552] random: crng init done [ 9.266048] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.525730] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.530453] printk: systemd: 26 output lines suppressed due to ratelimiting [ 10.764346] SELinux: Disabled at runtime. [ 10.828451] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 10.835253] systemd[1]: Detected virtualization kvm. [ 10.837063] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.344215] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.347564] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.353146] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.357484] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.361164] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.368135] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.376143] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-f[ 11.460965] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS s-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.757992] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 11.995591] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.018906] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.057335] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.078476] EDAC sbridge: Ver: 1.1.2 [ 13.071880] Key type dns_resolver registered [ 13.382238] NFS: Registering the id_resolver key type [ 13.384514] Key type id_resolver registered [ 13.386720] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg246-client login: [ 90.960544] libcfs: loading out-of-tree module taints kernel. [ 91.441321] Key type ._llcrypt registered [ 91.444060] Key type .llcrypt registered [ 92.114971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 92.144904] alg: No test for adler32 (adler32-zlib) [ 94.016741] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 95.211645] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 95.438138] hrtimer: interrupt took 2116638 ns [ 96.999697] Key type lgssc registered [ 99.548872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 255.172388] Lustre: Mounted lustre-client [ 260.909733] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 280.160935] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing check_logdir /tmp/testlogs/ [ 281.059329] Lustre: lustre-OST0000-osc-ffff9f14d018d000: disconnect after 23s idle [ 286.716698] Lustre: DEBUG MARKER: oleg246-client.virtnet: executing yml_node [ 291.361953] Lustre: DEBUG MARKER: Client: 2.17.54.115 [ 294.373578] Lustre: DEBUG MARKER: MDS: 2.17.54.115 [ 297.248719] Lustre: DEBUG MARKER: OSS: 2.17.54.115 [ 299.114705] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 22 02:43:55 EDT 2026 [ 317.425848] Lustre: Unmounted lustre-client [ 339.899588] Key type lgssc unregistered [ 340.375833] LNet: 8417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 340.392232] LNetError: 8417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 340.415966] LNet: Removed LNI 192.168.202.46@tcp [ 341.421936] Key type .llcrypt unregistered [ 341.425394] Key type ._llcrypt unregistered [ 373.759082] Key type ._llcrypt registered [ 373.761616] Key type .llcrypt registered [ 373.833688] LNet: 8998:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 373.833688] mask usage: [+|-] ... [ 374.127186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 374.138894] alg: No test for adler32 (adler32-zlib) [ 375.096299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 375.144845] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 375.170983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 379.576731] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 379.923171] LNet: 9303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.928216] LNetError: 9303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 379.941099] LNet: Removed LNI 192.168.202.46@tcp [ 380.535462] Key type .llcrypt unregistered [ 380.537540] Key type ._llcrypt unregistered [ 388.261749] Key type ._llcrypt registered [ 388.263932] Key type .llcrypt registered [ 389.198958] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:45:25 (1787381125) [ 390.266027] LNet: 9677:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 390.266027] mask usage: [+|-] ... [ 390.427435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 390.444798] alg: No test for adler32 (adler32-zlib) [ 391.398707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.471719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 391.487553] LNet: 9715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 391.502524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 391.535901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 391.548197] LNet: 9722:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 398.942896] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:45:35 (1787381135) [ 399.222463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.249111] LNet: 10300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 399.280440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.373205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 405.269303] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:45:41 (1787381141) [ 405.469832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.519903] LNet: 10886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.535667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.606406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 413.148424] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:45:49 (1787381149) [ 413.480811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.501522] LNet: 11471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.523842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.649277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 420.610755] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:45:57 (1787381157) [ 420.892237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.938710] LNet: 12056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.974773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.090272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 421.599666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 429.385878] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:46:05 (1787381165) [ 429.503555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.561254] LNet: 12641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.596813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.698260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 430.190918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 430.682551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 438.664621] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:46:14 (1787381174) [ 438.847632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.928741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.130776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 447.355177] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:46:23 (1787381183) [ 447.810753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.854703] LNet: 13829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.863245] LNet: 13829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 447.892634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.013530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 448.086493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.696404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 448.730891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 449.237746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 449.280152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.925524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 450.002549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 450.586083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 450.654399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 451.112139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 451.155294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 451.864771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 451.912976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 452.452851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 452.483719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 452.710754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 452.738909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 453.041951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 453.070082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 453.542790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 453.602384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 454.043397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 454.069198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 454.364378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.395501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 461.168171] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:46:37 (1787381197) [ 462.018731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.055496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.108939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 462.131697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 472.502416] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:46:47 (1787381207) [ 473.057232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.135321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.262395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 473.326367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 482.427118] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:46:58 (1787381218) [ 482.645992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.678141] LNet: 15682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.684471] LNet: 15682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 482.747937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.895325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 483.007535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 490.011738] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:47:06 (1787381226) [ 490.577916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.658122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.812522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 490.868546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 499.530058] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:47:15 (1787381235) [ 499.923756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.957794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.034601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 500.068073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 506.967669] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:47:23 (1787381243) [ 507.080514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.160043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.259118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 507.350549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 515.321130] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:47:31 (1787381251) [ 515.817717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.868857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.023723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.632666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 523.928626] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:47:40 (1787381260) [ 524.164682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.197424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.447890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.757263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.051983] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:47:49 (1787381269) [ 534.201227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.277722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.430521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 534.738118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 543.852560] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:47:59 (1787381279) [ 543.955119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.995652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.786116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.139610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 554.200377] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:48:10 (1787381290) [ 554.527205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.580939] LNet: 20518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.588531] LNet: 20518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 554.618570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.735512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 554.799267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 563.050191] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:48:19 (1787381299) [ 563.444142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.476888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.544398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 563.585834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 572.038296] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:48:28 (1787381308) [ 572.158275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.246832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.377226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 580.026916] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:48:35 (1787381315) [ 580.733428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.796125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.952984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 588.732445] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:48:44 (1787381324) [ 588.957405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.991945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.146752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 590.006375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 599.403305] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:48:55 (1787381335) [ 599.879861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.933818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.031343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 600.056711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 608.679202] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:49:04 (1787381344) [ 608.959175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.035670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.196957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 609.253570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 617.969378] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:49:13 (1787381353) [ 618.091633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.136977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.222575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 618.296492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 626.709480] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:49:22 (1787381362) [ 627.111315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.214912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.366321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 627.447070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 636.528707] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:49:32 (1787381372) [ 636.761540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.873932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.984368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 637.099969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 637.135739] LNetError: 25837:0:(peer.c:2234:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 637.228882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 637.276620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 645.827253] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:49:41 (1787381381) [ 646.042731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.160495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.271784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaKdAEWwHs [ 646.328594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaKdAEWwHs [ 646.395956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BaKdAEWwHs [ 654.529737] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:49:50 (1787381390) [ 654.906067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.968888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.074391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 655.134280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 655.211213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 662.265095] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:49:58 (1787381398) [ 662.732067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.796777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.872452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 666.344969] LNet: Message forwarding will be enabled in 2s [ 669.407350] LNet: Message forwarding enabled [ 670.011453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 670.040096] LNet: Message forwarding will stop in 2s to 6s [ 674.759635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 675.478266] LNet: Message forwarding disabled [ 677.587058] LNet: Message forwarding will be enabled in 2s [ 681.644964] LNet: Message forwarding enabled [ 685.112931] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:50:21 (1787381421) [ 685.488237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.517731] LNet: 28284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.528447] LNet: 28284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 687.116758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.225215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 695.848773] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:50:31 (1787381431) [ 695.972969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.015828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.102635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 696.123438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 696.154592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 696.185777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 696.231252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 696.271793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 696.315876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 696.391341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 696.460694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 696.517300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 696.599184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 696.634918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 696.662466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 696.695352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 696.719939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 696.750599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 696.781721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 696.815685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 696.843212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 696.860918] LNetError: 28910:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 704.577309] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:50:40 (1787381440) [ 704.712684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.799111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.934394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 713.676982] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:50:49 (1787381449) [ 716.555980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg246-server [ 718.854824] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:50:55 (1787381455) [ 719.339087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.429763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.489954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.524460] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 721.565810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[47-51]@tcp [ 721.628402] LNet: 30627:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 721.682221] LNetError: 30421:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.47@tcp has gone from up to down [ 721.777060] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 722.722118] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 722.732138] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 723.748942] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.48@tcp is being used as a gateway but routing feature is not turned on [ 723.762397] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 725.817609] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 725.850741] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 729.875608] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:51:06 (1787381466) [ 729.953392] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.51@tcp is being used as a gateway but routing feature is not turned on [ 729.960773] LNetError: 30417:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 731.734830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg246-server [ 734.214467] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:51:09 (1787381469) [ 734.461695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.485811] LNetError: 31563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.527566] LNet: Removed LNI 192.168.202.46@tcp [ 735.594308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.706406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.757942] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 736.795957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[47-52/2]@tcp [ 736.827322] LNet: 31680:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 736.845415] LNet: 31680:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 736.858159] LNetError: 31571:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.47@tcp has gone from up to down [ 736.871876] LNetError: 31571:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 736.872965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[47-52/2]@tcp [ 745.103166] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:51:20 (1787381480) [ 745.245961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.287733] LNetError: 32266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.343598] LNet: Removed LNI 192.168.202.46@tcp [ 746.432033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.608022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 746.689283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.746960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.857775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.945245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.039799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.146252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 747.256828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.297279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.390559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.499193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.540932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.596740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 747.700762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.732320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.846304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.943850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.009607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.125944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 748.224713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.275558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.361753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.473855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.500363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.537720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 748.634455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.694203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.824438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.916830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.952990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.990796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 749.067049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.143738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.228349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 757.005209] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:51:33 (1787381493) [ 759.097854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg246-server [ 761.398535] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:51:37 (1787381497) [ 763.491838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg246-server [ 765.588351] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:51:41 (1787381501) [ 766.572563] Key type .llcrypt unregistered [ 766.577144] Key type ._llcrypt unregistered [ 773.506544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 773.900127] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 773.904551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.914132] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 776.804683] Key type ._llcrypt registered [ 776.825213] Key type .llcrypt registered [ 776.886750] LNet: 33956:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 776.886750] mask usage: [+|-] ... [ 777.145816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.185736] alg: No test for adler32 (adler32-zlib) [ 778.172680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.245661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.290796] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 779.401987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 779.450243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.125781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 782.143371] LNet: Removed LNI 10.1.2.3@tcp [ 782.219289] LNet: 34399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.221616] LNetError: 34399:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.232244] LNet: Removed LNI 192.168.202.46@tcp [ 782.965384] Key type .llcrypt unregistered [ 782.966899] Key type ._llcrypt unregistered [ 798.582767] Key type ._llcrypt registered [ 798.586497] Key type .llcrypt registered [ 799.735791] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:52:16 (1787381536) [ 800.284226] Key type .llcrypt unregistered [ 800.286113] Key type ._llcrypt unregistered [ 806.039207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.349570] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.364801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.377675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.909340] Key type ._llcrypt registered [ 808.911012] Key type .llcrypt registered [ 808.964709] LNet: 35438:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 808.964709] mask usage: [+|-] ... [ 809.112789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.127399] alg: No test for adler32 (adler32-zlib) [ 810.098609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.215158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.251978] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 811.055466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 811.087539] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.853424] LNet: 35897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.863267] LNetError: 35897:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 813.866638] LNet: Removed LNI 10.1.2.3@tcp [ 813.895121] LNet: Removed LNI 192.168.202.46@tcp [ 814.514717] Key type .llcrypt unregistered [ 814.519222] Key type ._llcrypt unregistered [ 826.354530] Key type ._llcrypt registered [ 826.358702] Key type .llcrypt registered [ 827.910876] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:52:43 (1787381563) [ 828.416588] Key type .llcrypt unregistered [ 828.418642] Key type ._llcrypt unregistered [ 834.804462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.114859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.119825] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.122720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.678151] Key type ._llcrypt registered [ 837.679594] Key type .llcrypt registered [ 837.715891] LNet: 36935:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 837.715891] mask usage: [+|-] ... [ 837.888603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 837.924761] alg: No test for adler32 (adler32-zlib) [ 838.915109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.981985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 839.028277] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 840.832743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 840.847215] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.811584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 842.832726] LNet: Removed LNI 10.1.2.3@tcp [ 842.849526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 842.890942] LNet: Removed LNI 10.1.2.31@tcp [ 842.915334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 842.945944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.631650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 843.649412] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 844.440661] LNet: 37818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.446929] LNetError: 37818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 844.452792] LNet: Removed LNI 10.1.2.31@tcp [ 844.463913] LNet: Removed LNI 10.1.2.3@tcp [ 845.138548] Key type .llcrypt unregistered [ 845.147340] Key type ._llcrypt unregistered [ 858.279806] Key type ._llcrypt registered [ 858.282646] Key type .llcrypt registered [ 859.420465] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:53:15 (1787381595) [ 859.742794] LNet: 38611:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 859.742794] mask usage: [+|-] ... [ 859.902753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.919978] alg: No test for adler32 (adler32-zlib) [ 860.977507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.059789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.091761] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 861.747473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.762210] LNet: 38772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.776418] LNetError: 38772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.790688] LNet: Removed LNI 192.168.202.46@tcp [ 861.817786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.903180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.46@tcp [ 861.938929] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 862.871339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.46@tcp [ 862.892665] LNet: Removed LNI 192.168.202.46@tcp [ 862.944452] LNet: 38907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.516497] Key type .llcrypt unregistered [ 863.519886] Key type ._llcrypt unregistered [ 878.824699] Key type ._llcrypt registered [ 878.827207] Key type .llcrypt registered [ 880.051156] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:53:36 (1787381616) [ 882.059315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg246-server [ 883.861723] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:53:40 (1787381620) [ 884.794433] Key type .llcrypt unregistered [ 884.796731] Key type ._llcrypt unregistered [ 892.544269] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.700144] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.719535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.749340] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 895.698218] Key type ._llcrypt registered [ 895.699422] Key type .llcrypt registered [ 895.746709] LNet: 40299:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 895.746709] mask usage: [+|-] ... [ 895.871626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.879649] alg: No test for adler32 (adler32-zlib) [ 896.908511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.964221] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 897.538252] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 899.813910] LNet: 40759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.817911] LNetError: 40759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.821862] LNet: Removed LNI 192.168.202.46@tcp1 [ 899.837779] LNet: Removed LNI 10.1.2.3@tcp2 [ 900.650528] Key type .llcrypt unregistered [ 900.652912] Key type ._llcrypt unregistered [ 916.045624] Key type ._llcrypt registered [ 916.047587] Key type .llcrypt registered [ 917.687562] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:54:13 (1787381653) [ 917.924228] LNet: 41563:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 917.924228] mask usage: [+|-] ... [ 918.115247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.128775] alg: No test for adler32 (adler32-zlib) [ 919.175298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.237351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.279228] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 920.295522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.318521] LNet: 41731:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.324793] LNetError: 41731:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.342398] LNet: Removed LNI 192.168.202.46@tcp [ 920.358843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.424567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 920.468452] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 926.636024] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:54:23 (1787381663) [ 927.100759] LNet: 42322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.111729] LNetError: 42322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.138350] LNet: Removed LNI 192.168.202.46@tcp [ 927.833788] Key type .llcrypt unregistered [ 927.836898] Key type ._llcrypt unregistered [ 933.809480] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 933.994734] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 934.007456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 934.020664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.826629] Key type ._llcrypt registered [ 936.828589] Key type .llcrypt registered [ 936.872841] LNet: 42601:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 936.872841] mask usage: [+|-] ... [ 937.066862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.079944] alg: No test for adler32 (adler32-zlib) [ 938.032247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.081916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.105484] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 938.803837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 938.839933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.002770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.052466] LNet: 43067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.061122] LNetError: 43067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.068308] LNet: Removed LNI 10.1.2.3@tcp [ 941.075831] LNet: Removed LNI 192.168.202.46@tcp [ 941.116285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.204337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 941.243898] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 941.259468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 941.796823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.826555] LNet: 43094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.830693] LNetError: 43094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.847213] LNetError: 43094:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 941.864177] LNet: Removed LNI 10.1.2.3@tcp [ 941.880520] LNet: Removed LNI 192.168.202.46@tcp [ 941.913522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.996632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 942.028184] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 942.032858] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 942.489617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 942.511950] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 943.500440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.530383] LNet: 43256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.550674] LNetError: 43256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 943.559393] LNetError: 43256:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 943.570555] LNet: Removed LNI 10.1.2.3@tcp [ 943.589475] LNet: Removed LNI 192.168.202.46@tcp [ 943.605838] LNet: Removed LNI 192.168.202.46@tcp2 [ 943.634504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.727538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 943.764808] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 943.768326] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 943.775399] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 952.095788] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:54:48 (1787381688) [ 952.332738] LNet: 43854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.338311] LNetError: 43854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.345959] LNetError: 43854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 952.352918] LNet: Removed LNI 10.1.2.3@tcp [ 952.354564] LNet: Removed LNI 192.168.202.46@tcp [ 952.364243] LNet: Removed LNI 192.168.202.46@tcp2 [ 952.906672] Key type .llcrypt unregistered [ 952.908980] Key type ._llcrypt unregistered [ 959.256797] Key type ._llcrypt registered [ 959.262537] Key type .llcrypt registered [ 959.307705] LNet: 44080:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 959.307705] mask usage: [+|-] ... [ 959.523889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.540361] alg: No test for adler32 (adler32-zlib) [ 960.527457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.580952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.602602] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 961.470571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.493240] LNet: 44249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.500626] LNetError: 44249:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.511933] LNet: Removed LNI 192.168.202.46@tcp [ 961.538942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.603917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 961.624867] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 962.262382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.285165] LNet: 44279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.290336] LNetError: 44279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.305438] LNet: Removed LNI 192.168.202.46@tcp [ 962.337919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.427771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 962.475268] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 962.969829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.994149] LNet: Removed LNI 192.168.202.46@tcp [ 963.044346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.092587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 963.107944] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 963.403027] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 963.592286] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 963.599576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 963.603486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 964.988570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.003434] LNet: 44380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.020927] LNet: 44380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 965.041803] LNetError: 44380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.046729] LNetError: 44380:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 965.084551] LNet: Removed LNI 192.168.202.46@tcp [ 965.111700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.178723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 965.223107] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 973.192236] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:55:09 (1787381709) [ 973.529539] LNet: 44977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.560655] LNetError: 44977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.580429] LNet: Removed LNI 192.168.202.46@tcp [ 974.641660] Key type .llcrypt unregistered [ 974.645693] Key type ._llcrypt unregistered [ 982.785345] Key type ._llcrypt registered [ 982.788661] Key type .llcrypt registered [ 982.867354] LNet: 45203:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 982.867354] mask usage: [+|-] ... [ 983.146393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.169500] alg: No test for adler32 (adler32-zlib) [ 984.189712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.238657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 984.268442] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 985.377708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.406433] LNet: 45377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.415938] LNetError: 45377:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.437861] LNet: Removed LNI 192.168.202.46@tcp [ 985.455128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.552422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 985.591377] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 994.227642] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:55:30 (1787381730) [ 994.455186] LNet: 45968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.476263] LNetError: 45968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 994.520111] LNet: Removed LNI 192.168.202.46@tcp [ 995.280252] Key type .llcrypt unregistered [ 995.287326] Key type ._llcrypt unregistered [ 1002.417065] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1002.582485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1002.590212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1002.597392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1005.184399] Key type ._llcrypt registered [ 1005.186990] Key type .llcrypt registered [ 1005.232046] LNet: 46247:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1005.232046] mask usage: [+|-] ... [ 1005.378506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1005.393501] alg: No test for adler32 (adler32-zlib) [ 1006.318595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.373791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1006.393282] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1007.255112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.263715] LNet: 46420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.266075] LNetError: 46420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.279964] LNet: Removed LNI 192.168.202.46@tcp [ 1007.296204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.347276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1007.364950] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1017.108143] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:55:52 (1787381752) [ 1017.504170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1017.532774] LNet: 47019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.546860] LNetError: 47019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.596134] LNet: Removed LNI 192.168.202.46@tcp [ 1017.643607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1017.859833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1027.055662] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:56:02 (1787381762) [ 1027.221893] LNet: 47593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.015501] Key type .llcrypt unregistered [ 1028.020408] Key type ._llcrypt unregistered [ 1034.779168] Key type ._llcrypt registered [ 1034.784598] Key type .llcrypt registered [ 1034.809913] LNet: 47827:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1034.809913] mask usage: [+|-] ... [ 1035.007497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.022674] alg: No test for adler32 (adler32-zlib) [ 1036.027718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.089729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.105758] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1036.989494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.028715] LNet: 47999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.035790] LNetError: 47999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.054511] LNet: Removed LNI 192.168.202.46@tcp [ 1037.072800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.182933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1037.248525] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1043.241657] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:56:19 (1787381779) [ 1043.546606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1043.580786] LNet: 48592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.591737] LNetError: 48592:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.610904] LNet: Removed LNI 192.168.202.46@tcp [ 1043.657541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.777517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1051.292632] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:56:27 (1787381787) [ 1053.183708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1054.896262] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:56:31 (1787381791) [ 1056.103803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1057.729534] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:56:34 (1787381794) [ 1058.949311] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1060.387714] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:56:37 (1787381797) [ 1061.699821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1063.349553] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:56:39 (1787381799) [ 1064.802461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1066.825681] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 02:56:43 (1787381803) [ 1067.213678] LNet: 50645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.051852] Key type .llcrypt unregistered [ 1068.055429] Key type ._llcrypt unregistered [ 1074.401562] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1074.589399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1074.604171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1074.621093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1076.231505] Key type ._llcrypt registered [ 1076.234950] Key type .llcrypt registered [ 1076.289643] LNet: 50920:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1076.289643] mask usage: [+|-] ... [ 1076.508173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1076.539614] alg: No test for adler32 (adler32-zlib) [ 1077.512444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.612189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1077.646880] LNet: Added LNI 192.168.202.46@tcp [8/512/0/180] [ 1077.655727] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1077.660398] LNet: 50958:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1077.797318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.840421] LNet: 50977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.860566] LNetError: 50977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1077.874096] LNet: Removed LNI 10.1.2.3@tcp [ 1077.878103] LNetError: 50977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.897403] LNet: Removed LNI 192.168.202.46@tcp [ 1077.941508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.049491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1078.079104] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1078.175373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.201934] LNet: 51001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.218402] LNetError: 51001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.237773] LNet: Removed LNI 192.168.202.46@tcp [ 1078.269338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.368479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1078.387228] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1078.391399] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.399658] LNet: 51009:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1078.473842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.502487] LNet: 51027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.510504] LNetError: 51027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1078.519742] LNet: Removed LNI 10.1.2.3@tcp [ 1078.522574] LNetError: 51027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.559437] LNet: Removed LNI 192.168.202.46@tcp [ 1078.586420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.661325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1078.699960] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1078.706657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.710330] LNet: 51035:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1078.883930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1078.902902] LNet: 51061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1078.910166] LNetError: 51061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1078.919237] LNet: Removed LNI 10.1.2.3@tcp [ 1078.926080] LNetError: 51061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.936928] LNet: Removed LNI 192.168.202.46@tcp [ 1078.951235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.021840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1079.040233] LNet: Added LNI 192.168.202.46@tcp [8/256/0/182] [ 1079.047234] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1079.146470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1079.169400] LNet: 51089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.177270] LNetError: 51089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1079.185962] LNet: Removed LNI 10.1.2.3@tcp [ 1079.188422] LNetError: 51089:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.199862] LNet: Removed LNI 192.168.202.46@tcp [ 1079.214642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.275916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1079.287296] LNet: Added LNI 192.168.202.46@tcp [8/256/0/182] [ 1079.290821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1079.292678] LNet: 51097:0:(api-ni.c:2914:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1079.352308] LNet: Removed LNI 10.1.2.3@tcp [ 1079.361877] LNet: Removed LNI 192.168.202.46@tcp [ 1080.010534] Key type .llcrypt unregistered [ 1080.016126] Key type ._llcrypt unregistered [ 1093.208312] Key type ._llcrypt registered [ 1093.214202] Key type .llcrypt registered [ 1095.049432] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:57:10 (1787381830) [ 1095.419206] LNet: 51912:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1095.419206] mask usage: [+|-] ... [ 1095.592537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.607233] alg: No test for adler32 (adler32-zlib) [ 1096.694495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1096.777268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1096.798711] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1097.761844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.46 [ 1097.788678] LNet: 52074:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1097.797838] LNet: Removed LNI 192.168.202.46@tcp [ 1104.849515] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:57:21 (1787381841) [ 1105.064645] LNet: 52644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.076371] LNetError: 52644:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.098267] LNet: Removed LNI 192.168.202.46@tcp [ 1105.819358] Key type .llcrypt unregistered [ 1105.820986] Key type ._llcrypt unregistered [ 1112.576909] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1112.925725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1114.711868] Key type ._llcrypt registered [ 1114.714331] Key type .llcrypt registered [ 1114.767989] LNet: 52918:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1114.767989] mask usage: [+|-] ... [ 1115.046540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.080074] alg: No test for adler32 (adler32-zlib) [ 1116.682122] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1124.189165] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:57:40 (1787381860) [ 1124.316041] LNet: 53544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.333865] LNetError: 53544:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1124.393137] LNet: Removed LNI 10.1.2.3@tcp [ 1125.271268] Key type .llcrypt unregistered [ 1125.274121] Key type ._llcrypt unregistered [ 1132.366099] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1132.604682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1132.616043] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1132.632888] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1135.746733] Key type ._llcrypt registered [ 1135.751931] Key type .llcrypt registered [ 1135.805287] LNet: 53825:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1135.805287] mask usage: [+|-] ... [ 1135.938782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.949737] alg: No test for adler32 (adler32-zlib) [ 1137.012178] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1143.632464] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:58:00 (1787381880) [ 1145.334556] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1147.009406] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:58:03 (1787381883) [ 1147.203761] LNet: 54734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.213990] LNetError: 54734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1147.231113] LNet: Removed LNI 10.1.2.3@tcp [ 1147.874494] Key type .llcrypt unregistered [ 1147.879158] Key type ._llcrypt unregistered [ 1154.052712] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1154.389895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1154.396439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1154.400415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1157.253744] Key type ._llcrypt registered [ 1157.255569] Key type .llcrypt registered [ 1157.295795] LNet: 55011:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1157.295795] mask usage: [+|-] ... [ 1157.427607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1157.459533] alg: No test for adler32 (adler32-zlib) [ 1158.478176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1158.551588] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1167.670501] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:58:24 (1787381904) [ 1168.233389] LNet: 55966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.238940] LNetError: 55966:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1168.269454] LNet: Removed LNI 10.1.2.3@tcp [ 1169.001390] Key type .llcrypt unregistered [ 1169.008302] Key type ._llcrypt unregistered [ 1174.598550] Key type ._llcrypt registered [ 1174.600895] Key type .llcrypt registered [ 1174.635482] LNet: 56189:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1174.635482] mask usage: [+|-] ... [ 1174.767427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1174.781381] alg: No test for adler32 (adler32-zlib) [ 1175.707391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1175.753222] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1183.125199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1183.693231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 1183.720816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 1183.753794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1183.767862] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1183.976504] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1184.013668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.037512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1184.060364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.081196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.106647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.125878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.153634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.182786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.195200] LNet: 56404:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.208892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.229204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.257527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.277828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.308648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.331684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.359881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.784611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 1184.824443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 1184.871703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1184.888939] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1185.090575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.118557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1185.149290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.179404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.218848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.274782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.297648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.327310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.345945] LNet: 56513:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.379498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.433642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.479676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.519481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.561290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.603258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.638451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.140485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 1186.156677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 1186.184700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1186.201057] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1186.206059] LNet: Skipped 1 previous similar message [ 1186.402806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.428355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1186.452246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.484259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.516453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.543712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.594497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.639728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.657657] LNet: 56622:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.674717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.715212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.753276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.786700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.822119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.861592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.885177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.267776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 1187.293460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 1187.313550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1187.535121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.576556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1187.628466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.665408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.691600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.737650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.772704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.808890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.845909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.880531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.919814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.945803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.976907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.004559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.026490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.470386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 1188.504757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 1188.542914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1188.556845] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1188.567417] LNet: Skipped 1 previous similar message [ 1188.714655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.731641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1188.756898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.777469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.798798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.828458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.853505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.875703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.889283] LNet: 56840:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.891762] LNet: 56840:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1188.899817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.925663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.949884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.970440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.992881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.014752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.045355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.609853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 1189.669191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 1189.720557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1190.000515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.049835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1190.096959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.136436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.177996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.202825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.234208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.259511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.285459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.302211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.314419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.336820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.368313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.399141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.425427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.686406] LNet: 56991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.694747] LNetError: 56991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.716836] LNet: Removed LNI 192.168.202.46@tcp [ 1191.076425] Key type .llcrypt unregistered [ 1191.079245] Key type ._llcrypt unregistered [ 1205.970570] Key type ._llcrypt registered [ 1205.973422] Key type .llcrypt registered [ 1207.374661] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:59:03 (1787381943) [ 1208.464544] Key type .llcrypt unregistered [ 1208.466704] Key type ._llcrypt unregistered [ 1214.794183] Key type ._llcrypt registered [ 1214.798492] Key type .llcrypt registered [ 1214.861476] LNet: 58039:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1214.861476] mask usage: [+|-] ... [ 1215.070448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.090553] alg: No test for adler32 (adler32-zlib) [ 1216.079749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.108609] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1223.769451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1236.335545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.382379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1236.412130] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1238.093885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 1238.129870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 1 [ 1238.184579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 1238.225158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 1 [ 1238.262872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_interrupt -r 1 [ 1238.293974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 1 [ 1238.333799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_interrupt -r 1 [ 1238.371205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 1 [ 1238.412741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1238.432024] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1238.439623] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1238.617648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.640617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1238.667377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.698365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.727448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.753967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.778445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.792311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.800307] LNet: 58468:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.811756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.834952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.856108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.881817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.903559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.926775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.956734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.493757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 1239.530726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 1 [ 1239.600364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 1239.639119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 1 [ 1239.662597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_dropped -r 1 [ 1239.683890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 1 [ 1239.718175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_dropped -r 1 [ 1239.749534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 1 [ 1239.789857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1239.815396] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1239.821134] LNet: Skipped 2 previous similar messages [ 1240.046970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.070935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1240.096356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.119889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.154676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.184296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.235031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.295690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.318810] LNet: 58590:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.340757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.371708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.401816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.436694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.473971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.506766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.538729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.097157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 1241.142944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 1 [ 1241.170716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 1241.206724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 1 [ 1241.237962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_aborted -r 1 [ 1241.279949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 1 [ 1241.326564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_aborted -r 1 [ 1241.363405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 1 [ 1241.412571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1241.437673] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1241.447382] LNet: Skipped 2 previous similar messages [ 1241.730561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.772739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1241.821566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.856609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.948189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.984786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.032900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.082994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.106395] LNet: 58711:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.136171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.165742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.194491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.237671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1242.276699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1242.319654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.356867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.851213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 1242.874478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 1 [ 1242.899668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 1242.924974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 1 [ 1242.952985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_no_route -r 1 [ 1242.981327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 1 [ 1243.020353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_no_route -r 1 [ 1243.054374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 1 [ 1243.092020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1243.369082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.396892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1243.437772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.460873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.488222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.513533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.538642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.577228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.619551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.668835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.704551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.731262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.755700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.787493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.848306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1244.383786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 1244.404866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 1 [ 1244.457157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 1244.536242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 1 [ 1244.599384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_timeout -r 1 [ 1244.656811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 1 [ 1244.717374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_timeout -r 1 [ 1244.790079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 1 [ 1244.826206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1244.855906] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1244.865088] LNet: Skipped 5 previous similar messages [ 1245.141121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.166970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1245.195191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.215648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.252260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.278819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.327365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.351933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.384529] LNet: 58954:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.391743] LNet: 58954:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1245.409277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.432881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.472360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.522737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.555213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.602193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.637819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1246.096302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 1246.122730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_error -r 1 [ 1246.158282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 1246.192363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_error -r 1 [ 1246.225904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_error -r 1 [ 1246.263172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_error -r 1 [ 1246.291326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_error -r 1 [ 1246.322149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_error -r 1 [ 1246.342557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1246.602197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.671916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1246.706765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.742247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.787932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.835880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.856250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.900175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.937902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.963152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.982730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.009128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.041702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1247.085140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1247.121348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1253.494301] LNet: 59151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.502416] LNetError: 59151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.515594] LNet: Removed LNI 192.168.202.46@tcp [ 1253.528273] LNet: Removed LNI 192.168.202.46@tcp1 [ 1254.038859] Key type .llcrypt unregistered [ 1254.041383] Key type ._llcrypt unregistered [ 1270.995717] Key type ._llcrypt registered [ 1271.003414] Key type .llcrypt registered [ 1272.020588] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 03:00:08 (1787382008) [ 1273.025449] Key type .llcrypt unregistered [ 1273.029650] Key type ._llcrypt unregistered [ 1281.064506] Key type ._llcrypt registered [ 1281.068703] Key type .llcrypt registered [ 1281.107780] LNet: 60186:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1281.107780] mask usage: [+|-] ... [ 1281.350597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1281.381447] alg: No test for adler32 (adler32-zlib) [ 1282.471502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1282.512840] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1289.879234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1290.535959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 1290.606630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 1290.655677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1290.672857] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1290.839521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.864581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1290.892482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.917419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.934916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.959586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.982871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.004506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.020160] LNet: 60400:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.046665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.082240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.103315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.128602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.156626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.187592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.221833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.771108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 1291.795459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 1291.816844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1291.837745] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1292.076557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.105820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1292.130283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.162900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.187536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.216526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.250359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.280245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.295745] LNet: 60509:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.313703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.343449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.375163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.407529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.428420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.449884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.470451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1293.034442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 1293.089619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 1293.142439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1293.167875] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1293.465153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1293.490331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1293.520393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.570374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.591744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.612355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.640070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.665466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.683702] LNet: 60619:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.704764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.751464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.793664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.820989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.851427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.898299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.930400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.177854] LNet: 60660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.191411] LNetError: 60660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.212893] LNet: Removed LNI 192.168.202.46@tcp [ 1294.638459] Key type .llcrypt unregistered [ 1294.639655] Key type ._llcrypt unregistered [ 1311.392743] Key type ._llcrypt registered [ 1311.398447] Key type .llcrypt registered [ 1312.746760] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 03:00:48 (1787382048) [ 1313.600450] Key type .llcrypt unregistered [ 1313.602865] Key type ._llcrypt unregistered [ 1320.605971] Key type ._llcrypt registered [ 1320.611543] Key type .llcrypt registered [ 1320.701638] LNet: 61697:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1320.701638] mask usage: [+|-] ... [ 1320.831706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.867434] alg: No test for adler32 (adler32-zlib) [ 1321.867955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1321.921494] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1329.877917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1338.814968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.858951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1338.882326] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1340.462295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 1340.496128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 1 [ 1340.535219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 1340.565864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 1 [ 1340.600822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_dropped -r 1 [ 1340.632749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 1 [ 1340.660218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_dropped -r 1 [ 1340.724530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 1 [ 1340.798780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1340.827391] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1340.841207] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 1341.027837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.055184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1341.088494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.113839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.142386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.166414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.193749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.221709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.237066] LNet: 62139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.251949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.274994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.314822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.339320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.368442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.400555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.436622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.473198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.973644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 1342.013130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_error -r 1 [ 1342.048104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 1342.095453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_error -r 1 [ 1342.128908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_error -r 1 [ 1342.157176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_error -r 1 [ 1342.186773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_error -r 1 [ 1342.226549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_error -r 1 [ 1342.278066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1342.294828] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1342.300356] LNet: Skipped 2 previous similar messages [ 1342.540883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.571886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1342.609742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.651232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.686969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.716148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.740371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.777179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.804286] LNet: 62262:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.824826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.856658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.889659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.943520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.977782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.023423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.077625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.135153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.732183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 1343.783431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_timeout -r 1 [ 1343.844043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 1343.908526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_timeout -r 1 [ 1343.947459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_timeout -r 1 [ 1344.026317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_timeout -r 1 [ 1344.081537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_timeout -r 1 [ 1344.116601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_timeout -r 1 [ 1344.138250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.146@tcp [ 1344.154209] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1344.377738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.406962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1344.435410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.460734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1344.486668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1344.519958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1344.548822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1344.589173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1344.603442] LNet: 62386:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1344.618787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1344.640713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1344.692661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1344.725143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1344.753415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1344.775429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1344.803943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.831290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.510380] LNet: 62463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.520827] LNetError: 62463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.527460] LNet: Removed LNI 192.168.202.46@tcp [ 1349.550330] LNet: Removed LNI 192.168.202.46@tcp1 [ 1349.871362] Key type .llcrypt unregistered [ 1349.874361] Key type ._llcrypt unregistered [ 1366.233408] Key type ._llcrypt registered [ 1366.234956] Key type .llcrypt registered [ 1367.516547] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 03:01:43 (1787382103) [ 1369.002961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1370.889659] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 03:01:47 (1787382107) [ 1371.832545] Key type .llcrypt unregistered [ 1371.836912] Key type ._llcrypt unregistered [ 1379.004992] Key type ._llcrypt registered [ 1379.007322] Key type .llcrypt registered [ 1379.037710] LNet: 63792:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1379.037710] mask usage: [+|-] ... [ 1379.147763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1379.163846] alg: No test for adler32 (adler32-zlib) [ 1380.117316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1380.141804] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1387.350105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1388.099799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 1388.142901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 1388.179262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1388.225029] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1388.321236] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1388.465831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1388.493414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1388.549627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1388.608994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1388.642909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1388.684388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1388.721521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1388.755679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1388.777359] LNet: 64007:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1388.792505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1388.818603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1388.844780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1388.872702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1388.898259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1388.932751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1389.197710] LNet: 64046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.210537] LNetError: 64046:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.231474] LNet: Removed LNI 192.168.202.46@tcp [ 1389.610567] Key type .llcrypt unregistered [ 1389.616737] Key type ._llcrypt unregistered [ 1410.118559] Key type ._llcrypt registered [ 1410.124950] Key type .llcrypt registered [ 1410.171772] LNet: 64496:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1410.171772] mask usage: [+|-] ... [ 1410.348703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1410.380167] alg: No test for adler32 (adler32-zlib) [ 1411.357684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1411.389674] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1420.825975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1431.464142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.543238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1431.580486] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1432.819626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 1432.849985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e network_timeout -r 1 [ 1432.892442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 1432.955545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e network_timeout -r 1 [ 1433.028824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e network_timeout -r 1 [ 1433.078268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e network_timeout -r 1 [ 1433.136200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e network_timeout -r 1 [ 1433.181374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e network_timeout -r 1 [ 1433.228947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1433.254047] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1433.519726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.562792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1433.611379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.642144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.678532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.697574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1433.733460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1433.794123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1433.822042] LNet: 64926:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1433.851938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1433.883821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1433.952477] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1433.962616] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 1433.975054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.036994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.074108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.105610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1439.760456] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1440.074502] LNet: 65000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.080306] LNetError: 65000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.089233] LNet: Removed LNI 192.168.202.46@tcp [ 1440.107093] LNet: Removed LNI 192.168.202.46@tcp1 [ 1440.384647] Key type .llcrypt unregistered [ 1440.391059] Key type ._llcrypt unregistered [ 1459.664925] Key type ._llcrypt registered [ 1459.667430] Key type .llcrypt registered [ 1460.790341] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 03:03:17 (1787382197) [ 1461.206225] LNet: 65803:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1461.206225] mask usage: [+|-] ... [ 1461.307165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1461.314372] alg: No test for adler32 (adler32-zlib) [ 1462.211808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.248265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.266624] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1462.990648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.024319] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1463.953532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 1464.032565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1464.072575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1464.099784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1464.138379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1464.167766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1464.197853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 1464.216622] LNetError: 66102:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 1464.288032] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1464.289532] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1466.335362] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1466.723282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1470.431454] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1471.360640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1471.390817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1471.443611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.458854] LNet: 66209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.462813] LNetError: 66209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.479508] LNet: Removed LNI 192.168.202.46@tcp [ 1472.482844] LNetError: 66209:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.506217] LNet: Removed LNI 192.168.202.46@tcp1 [ 1472.535384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.605777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.637234] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1473.286836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.310201] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1474.103859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 1474.162118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1474.220809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1474.273913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1474.314872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1474.356752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1474.399516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1474.470522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 1474.494105] LNetError: 66482:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 1474.659680] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1476.918131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.801711] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1480.811067] LNet: Skipped 1 previous similar message [ 1486.043398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1486.104745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1486.154895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1492.932474] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 03:03:49 (1787382229) [ 1493.098404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.136328] LNet: 67194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.152635] LNetError: 67194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.175146] LNet: Removed LNI 192.168.202.46@tcp [ 1494.200439] LNet: Removed LNI 192.168.202.46@tcp1 [ 1494.226933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.285335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1494.310066] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1494.829474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1494.847633] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1495.423768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 1495.456353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1495.475825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1495.494648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1495.527510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1495.546583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1495.575549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 0 [ 1495.662318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1496.287299] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.46@tcp [ 1496.289424] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1496.303264] LNet: Skipped 1 previous similar message [ 1511.327156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1511.572139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1511.629380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1511.691480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 500 [ 1511.858436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1518.696716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1518.759615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.836981] LNet: 67690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.843111] LNetError: 67690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1518.854107] LNetError: 67690:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1518.874942] LNet: Removed LNI 192.168.202.46@tcp [ 1518.912206] LNet: Removed LNI 192.168.202.46@tcp1 [ 1518.982969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.088489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1519.129495] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1520.136741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.161268] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1520.997220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 1521.062133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1521.155220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1521.229426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1521.328064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1521.396372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1521.465340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1521.507979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.46@tcp --health 0 [ 1522.079517] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1522.088032] LNet: Skipped 3 previous similar messages [ 1522.989851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1533.545241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1533.571695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1533.635630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1544.018849] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 03:04:39 (1787382279) [ 1544.510201] LNet: 68736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.516533] LNetError: 68736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.522553] LNetError: 68736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1544.528146] LNet: Removed LNI 192.168.202.46@tcp [ 1544.537386] LNet: Removed LNI 192.168.202.46@tcp1 [ 1545.195444] Key type .llcrypt unregistered [ 1545.197202] Key type ._llcrypt unregistered [ 1552.624919] Key type ._llcrypt registered [ 1552.635539] Key type .llcrypt registered [ 1552.692085] LNet: 68960:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1552.692085] mask usage: [+|-] ... [ 1552.917699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1552.982691] alg: No test for adler32 (adler32-zlib) [ 1554.005740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1554.043716] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1564.166536] Lustre: DEBUG MARKER: Initial discovery [ 1564.371843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 1566.969299] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1567.173698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1567.250265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 1567.293526] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1567.314744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1567.371141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.146@tcp -r 1 -m GET -l 3 [ 1567.391579] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1567.408544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.146@tcp -r 1 -m GET -e local_error [ 1567.453853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.146@tcp -r 1 -m PUT -l 6 [ 1569.188220] Lustre: DEBUG MARKER: Force oleg246-client.virtnet to discover 192.168.202.146@tcp (in background) [ 1569.258935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.146@tcp [ 1571.342816] Lustre: DEBUG MARKER: Force oleg246-server to discover 192.168.202.46@tcp [ 1572.384047] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1573.571449] Lustre: DEBUG MARKER: Wait for 69491 [ 1575.137138] Lustre: DEBUG MARKER: Finished wait on 69491 [ 1575.189426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1575.221911] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1575.244745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1575.506369] LNet: 69949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.510903] LNetError: 69949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.525955] LNet: Removed LNI 192.168.202.46@tcp [ 1575.865715] Key type .llcrypt unregistered [ 1575.868989] Key type ._llcrypt unregistered [ 1593.133882] Key type ._llcrypt registered [ 1593.137651] Key type .llcrypt registered [ 1594.763828] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 03:05:30 (1787382330) [ 1595.459696] Key type .llcrypt unregistered [ 1595.462785] Key type ._llcrypt unregistered [ 1601.749931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1601.916795] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1601.922360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1601.925225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1604.510929] Key type ._llcrypt registered [ 1604.513535] Key type .llcrypt registered [ 1604.557566] LNet: 70997:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1604.557566] mask usage: [+|-] ... [ 1604.775656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1604.809591] alg: No test for adler32 (adler32-zlib) [ 1605.739287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.801789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.842301] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1606.477971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1606.509347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1614.522551] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 03:05:51 (1787382351) [ 1614.853625] LNet: 72031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.858351] LNetError: 72031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1614.867617] LNet: Removed LNI 10.1.2.3@tcp [ 1614.878191] LNet: Removed LNI 192.168.202.46@tcp [ 1615.411261] Key type .llcrypt unregistered [ 1615.412248] Key type ._llcrypt unregistered [ 1623.318221] Key type ._llcrypt registered [ 1623.321590] Key type .llcrypt registered [ 1623.367692] LNet: 72280:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1623.367692] mask usage: [+|-] ... [ 1623.650382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1623.674590] alg: No test for adler32 (adler32-zlib) [ 1624.634041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.717396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.766657] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1625.462836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1625.498750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1626.531451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.46@tcp [ 1627.615237] LNetError: Unexpected error -22 connecting to 192.168.202.46@tcp at host 192.168.202.46:988 [ 1632.909414] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 03:06:09 (1787382369) [ 1633.077418] LNet: 73161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.089024] LNetError: 73161:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1633.099037] LNet: Removed LNI 10.1.2.3@tcp [ 1633.118955] LNet: Removed LNI 192.168.202.46@tcp [ 1633.957458] Key type .llcrypt unregistered [ 1633.961175] Key type ._llcrypt unregistered [ 1641.294270] Key type ._llcrypt registered [ 1641.297235] Key type .llcrypt registered [ 1641.349310] LNet: 73393:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1641.349310] mask usage: [+|-] ... [ 1641.606532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.625272] alg: No test for adler32 (adler32-zlib) [ 1642.543182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.608360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1642.629911] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1643.418678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1643.436330] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 1644.139608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.46@tcp1 --nid 192.168.202.46@tcp2 [ 1647.973636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp1 192.168.202.46@tcp2 [ 1647.991416] LNetError: 73997:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.46@tcp1 to 192.168.202.46@tcp2 - Route table may be misconfigured [ 1648.015873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp2 192.168.202.46@tcp1 [ 1654.581563] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 03:06:30 (1787382390) [ 1654.694886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.727072] LNet: 74562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.738717] LNetError: 74562:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.742617] LNet: Removed LNI 192.168.202.46@tcp1 [ 1655.783325] LNet: Removed LNI 192.168.202.46@tcp2 [ 1655.812396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.871389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.911978] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1656.695263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.718254] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1657.595777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 1657.645629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.46@tcp [ 1657.682773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp1 192.168.202.46@tcp1 [ 1657.746727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp -d 192.168.202.46@tcp -e network_timeout [ 1657.826336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -e network_timeout [ 1657.883239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -e network_timeout [ 1657.931829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -e network_timeout [ 1657.978689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1658.002770] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1658.024817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1658.154756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1658.257441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1658.911145] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1664.245545] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 03:06:40 (1787382400) [ 1664.319492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.353356] LNet: 75424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.361909] LNetError: 75424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.366326] LNetError: 75424:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1664.371710] LNet: Removed LNI 192.168.202.46@tcp [ 1665.392363] LNet: Removed LNI 192.168.202.46@tcp1 [ 1665.448444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.545274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1665.847466] LNet: 75469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.138512] Key type .llcrypt unregistered [ 1666.140304] Key type ._llcrypt unregistered [ 1677.010370] Key type ._llcrypt registered [ 1677.013101] Key type .llcrypt registered [ 1678.022449] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 03:06:54 (1787382414) [ 1678.314388] LNet: 76240:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1678.314388] mask usage: [+|-] ... [ 1678.495617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1678.522951] alg: No test for adler32 (adler32-zlib) [ 1679.491607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.089121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1682.589795] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 03:06:59 (1787382419) [ 1682.785459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.820268] LNet: 76563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.844714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.930668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.955249] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1683.712407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1683.741831] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1684.472433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1684.522253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp1 [ 1684.569384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp1 [ 1691.283669] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 03:07:07 (1787382427) [ 1692.868940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg246-server [ 1694.752724] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 03:07:11 (1787382431) [ 1696.668715] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg246-server [ 1698.525674] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 03:07:14 (1787382434) [ 1699.932898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg246-server [ 1701.657301] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 03:07:18 (1787382438) [ 1703.244859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg246-server [ 1704.960736] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 03:07:21 (1787382441) [ 1706.762277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg246-server [ 1708.635594] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 03:07:25 (1787382445) [ 1710.772739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg246-server [ 1712.159443] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 03:07:28 (1787382448) [ 1714.095441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg246-server [ 1716.535499] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 03:07:32 (1787382452) [ 1718.886609] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg246-server [ 1720.152488] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 03:07:36 (1787382456) [ 1721.909504] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg246-server [ 1723.641376] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 03:07:40 (1787382460) [ 1723.908072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.950677] LNet: 80445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.962959] LNetError: 80445:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.972858] LNet: Removed LNI 192.168.202.46@tcp [ 1725.042149] LNet: Removed LNI 192.168.202.46@tcp1 [ 1725.076562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.180957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.226950] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1725.865481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1725.955853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1726.117604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.136568] LNet: 80593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.142818] LNetError: 80593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.150599] LNetError: 80593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1726.169790] LNet: Removed LNI 192.168.202.46@tcp [ 1726.183523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.225334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1726.246853] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1726.954370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1727.175705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.201121] LNetError: 80738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.216117] LNet: Removed LNI 192.168.202.46@tcp [ 1727.241491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.296927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.327406] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1728.075941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1728.189603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1728.353680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.384502] LNet: 80885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.397538] LNet: 80885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1729.448320] LNet: Removed LNI 192.168.202.46@tcp [ 1729.476177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.542789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1729.584558] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1730.494783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1730.720996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.743606] LNetError: 81030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.748416] LNetError: 81030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1730.772550] LNet: Removed LNI 192.168.202.46@tcp [ 1730.807345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.864523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.886187] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1731.672844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1731.808354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1731.988344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.030031] LNet: Removed LNI 192.168.202.46@tcp [ 1733.071449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.153538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1733.173098] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1733.896840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1734.118194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.147823] LNet: 81323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.152896] LNet: 81323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1735.205628] LNet: Removed LNI 192.168.202.46@tcp [ 1735.233790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.294589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1735.313481] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1736.131062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1736.228198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1736.466638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.494501] LNetError: 81470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.499487] LNetError: 81470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1737.578438] LNet: Removed LNI 192.168.202.46@tcp [ 1737.606237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.722248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1737.757098] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1738.865484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1739.088179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.138831] LNet: Removed LNI 192.168.202.46@tcp [ 1740.198293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.279641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.336357] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1741.238431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1741.357612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1741.565699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.642750] LNet: Removed LNI 192.168.202.46@tcp [ 1742.695595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.768962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1742.804631] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1743.416838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1743.609640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.630539] LNet: 81908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.638592] LNet: 81908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1744.677828] LNet: Removed LNI 192.168.202.46@tcp [ 1744.762126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.879019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1744.909321] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1745.687642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1745.788991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1746.048695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.077898] LNetError: 82055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.087558] LNetError: 82055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1746.101909] LNet: Removed LNI 192.168.202.46@tcp [ 1746.118842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.190455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1746.211083] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1746.945864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 1747.133266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.201238] LNet: Removed LNI 192.168.202.46@tcp [ 1748.246549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.329603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1748.350933] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1749.119549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1749.215470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.294228] LNet: Removed LNI 192.168.202.46@tcp [ 1749.327403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.380983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.413330] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1750.095873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1758.109902] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 03:08:14 (1787382494) [ 1758.500556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.549444] LNet: Removed LNI 192.168.202.46@tcp [ 1758.593207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.667732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1758.692323] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 1760.093916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.116279] LNet: 83175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.121554] LNet: 83175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1760.136269] LNet: Removed LNI 192.168.202.46@tcp231 [ 1760.150624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.225104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1760.258179] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 1760.264274] LNet: 83183:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1760.870933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1760.919443] LNet: Removed LNI 192.168.202.46@tcp231 [ 1760.943070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1760.994973] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 1762.166979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.208187] LNetError: 83328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.218257] LNetError: 83328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1762.232914] LNet: Removed LNI 192.168.202.46@tcp231 [ 1762.260352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.395614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1762.446406] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/0] [ 1762.453319] LNet: 83336:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1763.052136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1763.109733] LNet: Removed LNI 192.168.202.46@tcp231 [ 1763.150913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1763.196854] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/0] [ 1764.443762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.478546] LNet: Removed LNI 192.168.202.46@tcp231 [ 1764.510156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.612480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1764.643639] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/60] [ 1764.647511] LNet: 83487:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1765.151943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1765.192453] LNet: Removed LNI 192.168.202.46@tcp231 [ 1765.220531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1765.267957] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/60] [ 1766.665891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.689950] LNet: Removed LNI 192.168.202.46@tcp231 [ 1766.703483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.805432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1766.835629] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 1766.844264] LNet: 83638:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.307847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1767.359883] LNet: Removed LNI 192.168.202.46@tcp231 [ 1767.414551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1767.455659] LNet: Added LNI 192.168.202.46@tcp231 [8/256/0/180] [ 1775.679741] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 03:08:31 (1787382511) [ 1776.184314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.221383] LNet: Removed LNI 192.168.202.46@tcp231 [ 1776.248953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.360350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1776.385842] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 1777.352757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1777.399675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1777.451235] LNet: Removed LNI 192.168.202.46@tcp232 [ 1777.481752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1777.527232] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 1778.491355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1778.544687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1778.579360] LNet: Removed LNI 192.168.202.46@tcp232 [ 1778.594967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1778.660320] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 1779.754196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1779.823095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1779.891994] LNet: Removed LNI 192.168.202.46@tcp232 [ 1779.936126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1780.001831] LNet: Added LNI 192.168.202.46@tcp232 [8/256/0/180] [ 1781.035367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1781.091986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1781.155067] LNet: Removed LNI 192.168.202.46@tcp232 [ 1781.183462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1788.033027] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 03:08:44 (1787382524) [ 1789.132690] Key type .llcrypt unregistered [ 1789.136778] Key type ._llcrypt unregistered [ 1797.240770] Key type ._llcrypt registered [ 1797.246884] Key type .llcrypt registered [ 1797.345875] LNet: 85708:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1797.345875] mask usage: [+|-] ... [ 1797.621673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1797.643930] alg: No test for adler32 (adler32-zlib) [ 1798.734832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1798.770404] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1807.350357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1818.587470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.628596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1818.683645] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1820.504532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_interrupt -r 2 [ 1820.543430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 2 [ 1820.608848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_interrupt -r 2 [ 1820.676768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 2 [ 1820.747373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_interrupt -r 2 [ 1820.813917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_interrupt -r 2 [ 1820.907834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_interrupt -r 2 [ 1821.029399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_interrupt -r 2 [ 1821.115491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1821.172570] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1821.200940] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1821.640699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.685452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1821.733250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.788719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.829644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.871622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.929361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.975862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.995911] LNet: 86142:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.015275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1822.063212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1822.129959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.202932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1822.259496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1822.339515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1822.989154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_dropped -r 2 [ 1823.009643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 2 [ 1823.035582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_dropped -r 2 [ 1823.079982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 2 [ 1823.133336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_dropped -r 2 [ 1823.185985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_dropped -r 2 [ 1823.257332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_dropped -r 2 [ 1823.324191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_dropped -r 2 [ 1823.384190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1823.419642] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1823.639919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1823.665265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1823.690503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1823.714476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1823.744771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1823.777110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1823.817852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1823.877923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1823.927918] LNet: 86262:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1823.990951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1824.072601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1824.148324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1824.213378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1824.284493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1824.326433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1824.913844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_aborted -r 2 [ 1824.939724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 2 [ 1824.973876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_aborted -r 2 [ 1825.034584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 2 [ 1825.094220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_aborted -r 2 [ 1825.143396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_aborted -r 2 [ 1825.195427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_aborted -r 2 [ 1825.240475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_aborted -r 2 [ 1825.294817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1825.318520] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1825.631571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1825.664755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1825.696883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1825.731150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1825.759706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1825.857623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1825.932072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1825.996246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1826.021927] LNet: 86382:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.048407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1826.128755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1826.252946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1826.352408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1826.414935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1826.501952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1827.631715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_no_route -r 2 [ 1827.721746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 2 [ 1827.781557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_no_route -r 2 [ 1827.893775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 2 [ 1827.985911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_no_route -r 2 [ 1828.078625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_no_route -r 2 [ 1828.152495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_no_route -r 2 [ 1828.227877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_no_route -r 2 [ 1828.299529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1828.341459] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1828.801476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.842840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1828.882905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.974835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1829.043566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1829.106835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1829.146585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1829.192700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1829.220479] LNet: 86503:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.249387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1829.295912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.338253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.415638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.468608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.520809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1830.211666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e local_timeout -r 2 [ 1830.290804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 2 [ 1830.323789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e local_timeout -r 2 [ 1830.364079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 2 [ 1830.405207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e local_timeout -r 2 [ 1830.458114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e local_timeout -r 2 [ 1830.521038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e local_timeout -r 2 [ 1830.567664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e local_timeout -r 2 [ 1830.603708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1830.843349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1830.889742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1830.941658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1830.986756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.028985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.063650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.117804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.188204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.239442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.289930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.354731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.408894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.452440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.513185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.294450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp -m GET -e remote_dropped -r 2 [ 1832.337920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 2 [ 1832.381339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp -m GET -e remote_dropped -r 2 [ 1832.410613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 2 [ 1832.447709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp -m GET -e remote_dropped -r 2 [ 1832.479429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.46@tcp1 -d 192.168.202.146@tcp1 -m GET -e remote_dropped -r 2 [ 1832.548943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp -m GET -e remote_dropped -r 2 [ 1832.601935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.46@tcp1 -d 192.168.202.46@tcp1 -m GET -e remote_dropped -r 2 [ 1832.669478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.46@tcp 192.168.202.146@tcp [ 1832.698491] LNet: There was an unexpected network error while writing to 192.168.202.146: rc = -22 [ 1832.706235] LNet: Skipped 1 previous similar message [ 1832.718562] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 1833.043746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1833.076502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1833.106236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1833.167783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1833.213841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1833.251908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1833.294380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1833.343377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1833.361658] LNet: 86742:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1833.366782] LNet: 86742:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1833.396049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1833.445524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.517892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.553762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.629123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.709908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.868651] LNet: 86817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.874709] LNetError: 86817:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.883773] LNet: Removed LNI 192.168.202.46@tcp [ 1840.895440] LNet: Removed LNI 192.168.202.46@tcp1 [ 1841.302643] Key type .llcrypt unregistered [ 1841.305118] Key type ._llcrypt unregistered [ 1859.043197] Key type ._llcrypt registered [ 1859.044984] Key type .llcrypt registered [ 1860.333207] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 03:09:56 (1787382596) [ 1860.663708] LNet: 87624:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1860.663708] mask usage: [+|-] ... [ 1860.816616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.827257] alg: No test for adler32 (adler32-zlib) [ 1861.758403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.816138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1861.850445] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1862.843648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1862.878194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.46@tcp [ 1862.909208] LNetError: 87796:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 1862.977564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.46@tcp [ 1863.075406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1869.938450] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 03:10:06 (1787382606) [ 1870.175371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.216215] LNet: 88385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.232655] LNetError: 88385:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.256280] LNet: Removed LNI 192.168.202.46@tcp [ 1870.305975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.379476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.423077] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1871.222184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1871.240819] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 1872.164752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.46@tcp [ 1872.182558] LNetError: 88652:0:(api-ni.c:4400:lnet_ni_set_healthv()) manually adding local NI 192.168.202.46@tcp to recovery [ 1872.415295] LNet: 1 local NIs in recovery (showing 1): 192.168.202.46@tcp [ 1872.436318] LNet: There was an unexpected network error while writing to 192.168.202.46: rc = -22 [ 1873.489460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.46@tcp [ 1881.848168] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 03:10:18 (1787382618) [ 1883.551863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg246-server [ 1886.014598] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 03:10:22 (1787382622) [ 1888.099113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg246-server [ 1889.799220] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 03:10:26 (1787382626) [ 1891.792237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg246-server [ 1893.381270] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 03:10:29 (1787382629) [ 1893.602489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.622887] LNet: 90276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.628479] LNetError: 90276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.635554] LNet: Removed LNI 192.168.202.46@tcp [ 1893.650793] LNet: Removed LNI 192.168.202.46@tcp1 [ 1893.683143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.756827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1893.773589] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1894.431189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1894.492804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1894.577182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1894.727573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1894.778166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1894.849191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1901.214552] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 03:10:37 (1787382637) [ 1901.545722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.569981] LNet: 91001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.573926] LNetError: 91001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.579399] LNetError: 91001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1901.590024] LNet: Removed LNI 192.168.202.46@tcp [ 1901.607885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.682171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.707216] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1908.954408] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 03:10:45 (1787382645) [ 1909.127097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.169062] LNet: 91704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.178402] LNetError: 91704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1909.200068] LNet: Removed LNI 192.168.202.46@tcp [ 1909.228906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.320979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.381968] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1916.680959] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 03:10:52 (1787382652) [ 1918.050573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1920.771232] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 03:10:56 (1787382656) [ 1921.470844] LNet: 92739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.479567] LNetError: 92739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.488539] LNet: Removed LNI 192.168.202.46@tcp [ 1922.221555] Key type .llcrypt unregistered [ 1922.224129] Key type ._llcrypt unregistered [ 1928.563402] Key type ._llcrypt registered [ 1928.565672] Key type .llcrypt registered [ 1928.657666] LNet: 92968:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1928.657666] mask usage: [+|-] ... [ 1928.864346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1928.881246] alg: No test for adler32 (adler32-zlib) [ 1929.886353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1929.908434] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1937.219741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 1943.366928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.146@tcp [ 1943.694678] LNet: 93157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.706479] LNetError: 93157:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.723671] LNet: Removed LNI 192.168.202.46@tcp [ 1944.059434] Key type .llcrypt unregistered [ 1944.061185] Key type ._llcrypt unregistered [ 1955.823538] Key type ._llcrypt registered [ 1955.827369] Key type .llcrypt registered [ 1957.100256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1958.954576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1960.744449] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 03:11:37 (1787382697) [ 1962.323131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1963.964869] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 03:11:40 (1787382700) [ 1965.840643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg246-server [ 1967.926035] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 03:11:44 (1787382704) [ 1969.890071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg246-server [ 1972.071569] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 03:11:48 (1787382708) [ 1973.389620] LNet: 95208:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 1973.389620] mask usage: [+|-] ... [ 1974.143182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.192929] alg: No test for adler32 (adler32-zlib) [ 1975.481887] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 1975.868846] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 1977.567646] Key type lgssc registered [ 1979.415699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2003.320965] Key type lgssc unregistered [ 2003.733853] LNet: 96334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2003.743890] LNetError: 96334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2003.765366] LNet: Removed LNI 192.168.202.46@tcp [ 2004.764626] Key type .llcrypt unregistered [ 2004.766227] Key type ._llcrypt unregistered [ 2022.142566] Key type ._llcrypt registered [ 2022.144360] Key type .llcrypt registered [ 2023.778171] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 03:12:39 (1787382759) [ 2024.138765] LNet: 97131:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2024.138765] mask usage: [+|-] ... [ 2024.348646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2024.364477] alg: No test for adler32 (adler32-zlib) [ 2025.255099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.322673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2025.425780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2025.483762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2025.558853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2032.555756] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 03:12:48 (1787382768) [ 2032.699875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2032.726706] LNet: 97746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.773411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.823109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.870639] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2033.798862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2033.816542] LNet: 97893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.820641] LNetError: 97893:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.831837] LNet: Removed LNI 192.168.202.46@tcp [ 2033.849804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2033.903967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2033.937867] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2034.593086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2034.619602] LNetError: 97921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.655735] LNet: Removed LNI 192.168.202.46@tcp [ 2034.679287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2034.737317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2034.763066] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2042.161955] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 03:12:58 (1787382778) [ 2042.431691] LNet: 98511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.441607] LNet: 98511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2042.454990] LNetError: 98511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.474932] LNet: Removed LNI 192.168.202.46@tcp [ 2043.153418] Key type .llcrypt unregistered [ 2043.157934] Key type ._llcrypt unregistered [ 2049.844502] Key type ._llcrypt registered [ 2049.847387] Key type .llcrypt registered [ 2049.895882] LNet: 98744:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2049.895882] mask usage: [+|-] ... [ 2050.113242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.152442] alg: No test for adler32 (adler32-zlib) [ 2051.241401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.409725] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2051.416934] LNet: *** cfs_fail_loc=e003, val=100*** [ 2052.481562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 2052.514712] LNet: *** cfs_fail_loc=e003, val=100*** [ 2052.518318] LNet: Skipped 2 previous similar messages [ 2052.533268] Ping reply received with mapping: [ 2052.536474] found metadata reply NI[192.168.202.46@tcp] [ 2052.539682] 2e ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.541233] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.545422] 5a 5a 5a 5a 5a 5a 5a 5a [ 2052.686616] LNet: 98910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.694917] LNetError: 98910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.734873] LNet: Removed LNI 192.168.202.46@tcp [ 2054.612321] Key type .llcrypt unregistered [ 2054.614577] Key type ._llcrypt unregistered [ 2067.525335] Key type ._llcrypt registered [ 2067.532615] Key type .llcrypt registered [ 2069.283435] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 03:13:24 (1787382804) [ 2069.975638] Key type .llcrypt unregistered [ 2069.977598] Key type ._llcrypt unregistered [ 2078.945198] Key type ._llcrypt registered [ 2078.947159] Key type .llcrypt registered [ 2078.992247] LNet: 99907:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2078.992247] mask usage: [+|-] ... [ 2079.231901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.271960] alg: No test for adler32 (adler32-zlib) [ 2080.627892] LNetError: 99958:0:(api-ni.c:2779:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2081.137446] Key type .llcrypt unregistered [ 2081.140343] Key type ._llcrypt unregistered [ 2087.528696] Key type ._llcrypt registered [ 2087.531863] Key type .llcrypt registered [ 2089.278220] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 03:13:45 (1787382825) [ 2090.162429] Key type .llcrypt unregistered [ 2090.164071] Key type ._llcrypt unregistered [ 2098.310126] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2098.317757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2100.996173] Key type ._llcrypt registered [ 2101.000095] Key type .llcrypt registered [ 2101.048255] LNet: 100828:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2101.048255] mask usage: [+|-] ... [ 2101.132915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.143935] alg: No test for adler32 (adler32-zlib) [ 2102.060995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.157912] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2112.036561] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 03:14:08 (1787382848) [ 2112.442579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.490984] LNet: 101742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.501960] LNetError: 101742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2112.527948] LNet: Removed LNI 10.1.2.3@tcp [ 2112.550992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.504840] Key type .llcrypt unregistered [ 2113.507662] Key type ._llcrypt unregistered [ 2121.177865] Key type ._llcrypt registered [ 2121.180953] Key type .llcrypt registered [ 2121.225868] LNet: 101973:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2121.225868] mask usage: [+|-] ... [ 2121.331640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2121.340377] alg: No test for adler32 (adler32-zlib) [ 2122.306059] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2122.388702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 2122.572725] LNet: 102030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.577341] LNetError: 102030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2122.590211] LNet: Removed LNI 192.168.202.46@tcp [ 2123.103375] Key type .llcrypt unregistered [ 2123.105875] Key type ._llcrypt unregistered [ 2134.035722] Key type ._llcrypt registered [ 2134.038513] Key type .llcrypt registered [ 2135.144618] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 03:14:31 (1787382871) [ 2135.698578] Key type .llcrypt unregistered [ 2135.700659] Key type ._llcrypt unregistered [ 2143.768847] Key type ._llcrypt registered [ 2143.773596] Key type .llcrypt registered [ 2143.832958] LNet: 103016:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2143.832958] mask usage: [+|-] ... [ 2143.973883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.985395] alg: No test for adler32 (adler32-zlib) [ 2152.184157] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 03:14:48 (1787382888) [ 2152.470300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.524232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.599545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.635377] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2153.486458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.505537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2153.538649] LNet: Removed LNI 192.168.202.46@tcp [ 2153.815430] LNet: 103777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.242318] Key type .llcrypt unregistered [ 2154.245641] Key type ._llcrypt unregistered [ 2167.495638] Key type ._llcrypt registered [ 2167.498612] Key type .llcrypt registered [ 2168.842638] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 03:15:04 (1787382904) [ 2169.451527] LNet: 104542:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2169.451527] mask usage: [+|-] ... [ 2169.758679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2169.781692] alg: No test for adler32 (adler32-zlib) [ 2170.736287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.801709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.816308] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2171.656845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 2171.748220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.46@tcp [ 2171.848880] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2171.856820] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.863609] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.872975] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2171.880928] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2171.889026] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.895434] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 0 [ 2171.902769] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2171.905943] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.914308] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.928263] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.937430] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2171.943641] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2171.949936] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2171.960584] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2171, 0, 2221 [ 2171.967537] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2179.812167] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 03:15:15 (1787382915) [ 2180.533939] LNet: 105317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.543384] LNetError: 105317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.570344] LNet: Removed LNI 192.168.202.46@tcp [ 2181.857562] Key type .llcrypt unregistered [ 2181.863262] Key type ._llcrypt unregistered [ 2190.202631] Key type ._llcrypt registered [ 2190.204308] Key type .llcrypt registered [ 2190.235462] LNet: 105542:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2190.235462] mask usage: [+|-] ... [ 2190.411770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2190.441635] alg: No test for adler32 (adler32-zlib) [ 2191.297628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2191.316031] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2200.691205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 2210.909982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2210.942353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2210.978730] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 2212.209623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2212.421261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2212.436108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2212.454683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2214.865451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2214.894514] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2227.510563] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 03:16:03 (1787382963) [ 2227.884595] LNet: 106822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.887937] LNetError: 106822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.890802] LNet: Removed LNI 192.168.202.46@tcp [ 2228.895138] LNet: Removed LNI 192.168.202.46@tcp1 [ 2228.898134] LNetError: 106822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2228.904796] LNetError: 106822:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2228.952799] LNet: Removed LNI 10.1.2.3@tcp99 [ 2229.764442] Key type .llcrypt unregistered [ 2229.769777] Key type ._llcrypt unregistered [ 2237.741281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2237.744587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2239.371404] Key type ._llcrypt registered [ 2239.374170] Key type .llcrypt registered [ 2239.415078] LNet: 107079:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2239.415078] mask usage: [+|-] ... [ 2239.761743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2239.804977] alg: No test for adler32 (adler32-zlib) [ 2240.812610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.963275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.001472] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2242.376906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2242.444752] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2245.197444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2245.240949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.281379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2245.303473] LNetError: 107558:0:(peer.c:2234:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2245.356543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.423866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2245.867916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2245.922450] LNet: 107578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.926333] LNetError: 107578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2245.930767] LNet: Removed LNI 10.1.2.3@tcp [ 2246.950373] LNet: Removed LNI 192.168.202.46@tcp [ 2247.926322] Key type .llcrypt unregistered [ 2247.934436] Key type ._llcrypt unregistered [ 2262.789624] Key type ._llcrypt registered [ 2262.791036] Key type .llcrypt registered [ 2263.801341] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 03:16:40 (1787383000) [ 2264.247380] LNet: 108378:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2264.247380] mask usage: [+|-] ... [ 2264.452546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2264.470445] alg: No test for adler32 (adler32-zlib) [ 2265.463167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.557821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.595213] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2266.365397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.46@tcp [ 2266.448425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg246-client@tcp [ 2273.222981] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 03:16:49 (1787383009) [ 2273.304300] LNet: 109104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.307808] LNetError: 109104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.333316] LNet: Removed LNI 192.168.202.46@tcp [ 2274.121629] Key type .llcrypt unregistered [ 2274.127099] Key type ._llcrypt unregistered [ 2280.395767] Key type ._llcrypt registered [ 2280.397444] Key type .llcrypt registered [ 2280.614252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2280.644637] alg: No test for adler32 (adler32-zlib) [ 2281.632514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2281.667766] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2281.714072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2288.211692] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 03:17:04 (1787383024) [ 2289.626920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2291.235614] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 03:17:07 (1787383027) [ 2292.995699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2295.106792] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 03:17:11 (1787383031) [ 2295.402297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2295.437115] LNet: 110532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.446612] LNetError: 110532:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.467333] LNet: Removed LNI 192.168.202.46@tcp [ 2295.508383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.577467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2295.605983] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2296.280709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2296.307959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2302.981719] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 03:17:19 (1787383039) [ 2303.300574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.319089] LNet: 111242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.326592] LNetError: 111242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.341951] LNet: Removed LNI 192.168.202.46@tcp [ 2303.355462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.403379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2303.443994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2303.999347] Key type .llcrypt unregistered [ 2304.000850] Key type ._llcrypt unregistered [ 2314.823875] Key type ._llcrypt registered [ 2314.826610] Key type .llcrypt registered [ 2316.130491] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 03:17:32 (1787383052) [ 2316.439342] LNet: 112050:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2316.439342] mask usage: [+|-] ... [ 2316.557210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.570991] alg: No test for adler32 (adler32-zlib) [ 2317.537987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.595559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.621696] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2318.210141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2318.267620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2324.195186] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 03:17:40 (1787383060) [ 2324.386521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.406261] LNet: 112798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.409876] LNetError: 112798:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.479807] LNet: Removed LNI 192.168.202.46@tcp [ 2325.495753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2325.540872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2325.567694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2331.543175] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 03:17:47 (1787383067) [ 2331.676773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.701292] LNet: 113373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.751349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.829783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2332.254871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2332.305607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2332.417953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2332.983995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.029174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.123767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.142031] LNet: 113411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.171676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.299623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2333.729864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.779553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.854859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2334.347852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.368631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2340.664784] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 03:17:56 (1787383076) [ 2340.777190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2340.809163] LNet: 114005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.849734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.004204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.033604] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2342.043769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.074525] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2342.818245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2342.868598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2343.230077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.256898] LNetError: 114291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.287507] LNet: Removed LNI 192.168.202.46@tcp [ 2344.292753] LNetError: 114291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.324965] LNet: Removed LNI 192.168.202.46@tcp2 [ 2344.362724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.458623] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2344.464570] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2344.470945] LNet: 114297:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2345.317794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2345.524904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.551360] LNet: 114334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.556445] LNet: 114334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2346.591218] LNet: Removed LNI 192.168.202.46@tcp [ 2346.594845] LNetError: 114334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.603823] LNetError: 114334:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2346.618681] LNet: Removed LNI 192.168.202.46@tcp2 [ 2346.659726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.750809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2346.771816] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2347.446990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2347.462508] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2348.268261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2348.327493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2348.419176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2348.703756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.791215] LNet: Removed LNI 192.168.202.46@tcp [ 2350.822724] LNet: Removed LNI 192.168.202.46@tcp2 [ 2350.853458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.957707] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2350.972786] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2350.992221] LNet: 114625:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2351.364977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2351.557487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2351.598180] LNetError: 114663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.609307] LNetError: 114663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2351.614915] LNet: Removed LNI 192.168.202.46@tcp [ 2352.680684] LNet: Removed LNI 192.168.202.46@tcp2 [ 2352.700627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.780391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2352.802863] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2353.509937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2353.535145] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2354.293355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2354.361332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2354.567880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.593537] LNet: 114942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.601386] LNet: 114942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2355.615137] LNet: Removed LNI 192.168.202.46@tcp [ 2355.631243] LNet: Removed LNI 192.168.202.46@tcp2 [ 2355.658472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.714619] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2355.722500] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2355.726573] LNet: 114948:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.217288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2356.419909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.471160] LNet: Removed LNI 192.168.202.46@tcp [ 2357.483445] LNet: Removed LNI 192.168.202.46@tcp2 [ 2357.518434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.585837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2357.632922] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2358.366520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2358.391525] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2359.243837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2359.282545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2359.338561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2359.577800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.598623] LNetError: 115270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.604367] LNetError: 115270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2359.613721] LNet: Removed LNI 192.168.202.46@tcp [ 2360.683712] LNet: Removed LNI 192.168.202.46@tcp2 [ 2360.792296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.857329] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2360.862646] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2360.867875] LNet: 115276:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.265817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2361.428657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.440735] LNet: Removed LNI 192.168.202.46@tcp [ 2362.477083] LNet: Removed LNI 192.168.202.46@tcp2 [ 2362.522229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.600986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2362.625975] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2363.212961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2363.242726] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2364.207650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2364.234890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2364.302833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2364.657656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.727140] LNet: Removed LNI 192.168.202.46@tcp [ 2365.740221] LNet: Removed LNI 192.168.202.46@tcp2 [ 2365.764480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.856661] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2365.862811] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2365.866891] LNet: 115616:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.461357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2366.744944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.776163] LNet: Removed LNI 192.168.202.46@tcp [ 2367.788769] LNet: Removed LNI 192.168.202.46@tcp2 [ 2367.827289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.901090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2367.924497] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2368.542941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2368.569796] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2369.337838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2369.404386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2369.435605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2369.493782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2369.835490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2370.911154] LNet: Removed LNI 192.168.202.46@tcp [ 2371.944535] LNet: Removed LNI 192.168.202.46@tcp2 [ 2371.961272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.016782] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2372.020940] LNet: Added LNI 192.168.202.46@tcp2 [8/256/0/180] [ 2372.523332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2380.651366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2382.768540] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 03:18:38 (1787383118) [ 2382.872326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.904893] LNet: 116730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2382.913388] LNet: 116730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2382.924295] LNetError: 116730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2382.931075] LNetError: 116730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2382.945434] LNet: Removed LNI 192.168.202.46@tcp [ 2383.977034] LNet: Removed LNI 192.168.202.46@tcp2 [ 2384.016976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.412838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2384.429119] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2385.152429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2385.173225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2386.239298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.266405] LNet: Removed LNI 10.1.2.3@tcp [ 2386.281891] LNet: Removed LNI 192.168.202.46@tcp [ 2386.320748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.420835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2386.449678] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2386.459149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2387.094918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2387.121780] LNet: Removed LNI 192.168.202.46@tcp [ 2387.141597] LNet: Removed LNI 10.1.2.3@tcp [ 2393.093648] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 03:18:49 (1787383129) [ 2393.437776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2393.507231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2393.594982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2393.694550] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2393.714723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2393.723671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2396.166397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.204919] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2397.013357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2397.030841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2398.746696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.46@tcp [ 2398.781285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2398.873700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.46@tcp -m PUT -e local_timeout -r 1 [ 2398.933736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2399.020246] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2399.026974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2399.178163] LNetError: 117674:0:(lib-move.c:4714:lnet_parse()) 192.168.202.46@tcp, src 192.168.202.46@tcp: Dropping PUT (error -108 looking up sender) [ 2400.095735] LNet: Removed LNI 10.1.2.3@tcp [ 2401.129534] LNet: Removed LNI 192.168.202.46@tcp [ 2401.613388] Key type .llcrypt unregistered [ 2401.615872] Key type ._llcrypt unregistered [ 2412.413454] Key type ._llcrypt registered [ 2412.415262] Key type .llcrypt registered [ 2413.741823] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 03:19:09 (1787383149) [ 2413.973423] LNet: 118903:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2413.973423] mask usage: [+|-] ... [ 2414.086632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.101158] alg: No test for adler32 (adler32-zlib) [ 2415.054942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.378752] LNet: 118984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.852396] Key type .llcrypt unregistered [ 2415.862507] Key type ._llcrypt unregistered [ 2421.447322] Key type ._llcrypt registered [ 2421.449700] Key type .llcrypt registered [ 2421.478870] LNet: 119213:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2421.478870] mask usage: [+|-] ... [ 2421.710579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2421.729472] alg: No test for adler32 (adler32-zlib) [ 2422.656769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2422.687314] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2429.422786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 2429.680825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2430.531834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.146@tcp [ 2430.875677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2430.884138] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 2430.906893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2430.935550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2430.976599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2431.031778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2431.076328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2431.116807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2431.152670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2431.177940] LNet: 119434:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2431.231569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2431.264641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2431.324914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2431.386591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2431.428670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2431.466773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2432.263896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2432.626555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2432.661653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2432.685497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2432.738236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2432.813907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2432.893324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2432.953401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2433.029033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2433.065173] LNet: 119552:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2433.108758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2433.189384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2433.268277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2433.310698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2433.339698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2433.382941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2434.247314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.146@tcp [ 2434.485624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2434.519727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2434.555353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2434.585718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2434.623930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2434.674351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2434.736351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2434.804332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2434.830666] LNet: 119669:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2434.858739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2434.895912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2434.952252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2435.000950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2435.066322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2435.158172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2436.079612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2436.406344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2436.439674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2436.477621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2436.520574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2436.571946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2436.653100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2436.712978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2436.762485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2436.811941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2436.860822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2436.880437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2436.910990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2436.955913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2436.990612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2437.363643] LNet: 119832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.372476] LNetError: 119832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.395340] LNet: Removed LNI 192.168.202.46@tcp [ 2437.846931] Key type .llcrypt unregistered [ 2437.849481] Key type ._llcrypt unregistered [ 2455.120545] Key type ._llcrypt registered [ 2455.122762] Key type .llcrypt registered [ 2456.676340] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 03:19:52 (1787383192) [ 2456.845946] LNet: 120640:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2456.845946] mask usage: [+|-] ... [ 2456.991388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.007484] alg: No test for adler32 (adler32-zlib) [ 2458.034786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.461681] LNet: 120722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2459.177709] Key type .llcrypt unregistered [ 2459.179456] Key type ._llcrypt unregistered [ 2465.871520] Key type ._llcrypt registered [ 2465.876654] Key type .llcrypt registered [ 2465.942924] LNet: 120951:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2465.942924] mask usage: [+|-] ... [ 2466.124933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2466.152838] alg: No test for adler32 (adler32-zlib) [ 2467.174644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2467.195142] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2474.176716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.146@tcp [ 2482.709988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.769588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2482.801208] LNet: Added LNI 192.168.202.46@tcp1 [8/256/0/180] [ 2483.820957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2484.696515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.146@tcp [ 2484.930530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2484.982446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2485.031627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2485.076667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2485.148801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2485.208448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2485.311130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2485.355689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2485.403633] LNet: 121375:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.438425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2485.487837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2485.538293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2485.664391] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.146@tcp [ 2485.697245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2485.768728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2485.830378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.589483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2486.898770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2486.957313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2486.997879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2487.029460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2487.074223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2487.110320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2487.141771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.185994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.223680] LNet: 121492:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.259559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.319850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.373843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.433268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.475680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.526196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2488.472508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.146@tcp [ 2488.808040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2488.890392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2488.953720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2489.029634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2489.082181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2489.103202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2489.126822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2489.175801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2489.214316] LNet: 121610:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2489.253108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2489.283492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2489.314381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2489.360760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2489.395626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2489.453565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.448960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2491.092975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2491.173493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2491.262663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2491.327272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2491.408563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2491.450768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2491.499824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2491.575608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2491.600427] LNet: 121727:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2491.629523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2491.674322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2491.819736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2491.942738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2492.042417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2492.075380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2497.647856] LNet: 121808:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2497.653766] LNetError: 121808:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.662622] LNet: Removed LNI 192.168.202.46@tcp [ 2497.674819] LNet: Removed LNI 192.168.202.46@tcp1 [ 2498.052873] Key type .llcrypt unregistered [ 2498.056322] Key type ._llcrypt unregistered [ 2516.597077] Key type ._llcrypt registered [ 2516.598214] Key type .llcrypt registered [ 2518.002437] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 03:20:54 (1787383254) [ 2524.403510] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 03:21:01 (1787383261) [ 2530.472392] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 03:21:07 (1787383267) [ 2538.263224] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 03:21:14 (1787383274) [ 2545.838454] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 03:21:22 (1787383282) [ 2553.230208] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 03:21:29 (1787383289) [ 2562.125878] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 03:21:37 (1787383297) [ 2570.043361] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 03:21:46 (1787383306) [ 2577.724570] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 03:21:54 (1787383314) [ 2584.083732] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 03:22:00 (1787383320) [ 2590.744962] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 03:22:07 (1787383327) [ 2598.691454] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 03:22:15 (1787383335) [ 2605.086918] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 03:22:21 (1787383341) [ 2612.383453] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 03:22:28 (1787383348) [ 2621.023240] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 03:22:37 (1787383357) [ 2628.221693] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 03:22:44 (1787383364) [ 2636.652251] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 03:22:52 (1787383372) [ 2644.879814] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 03:23:01 (1787383381) [ 2645.102672] LNet: 132349:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2645.102672] mask usage: [+|-] ... [ 2645.324821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2645.335639] alg: No test for adler32 (adler32-zlib) [ 2646.314273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2646.482811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2646.514791] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2647.657683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2647.691807] LNet: 132522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2647.697683] LNetError: 132522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2647.712298] LNet: Removed LNI 192.168.202.46@tcp [ 2647.727551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2647.832755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2647.911654] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2655.530280] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 03:23:11 (1787383391) [ 2663.002200] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 03:23:19 (1787383399) [ 2669.813622] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 03:23:26 (1787383406) [ 2676.912765] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 03:23:33 (1787383413) [ 2684.381472] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 03:23:41 (1787383421) [ 2692.879714] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 03:23:49 (1787383429) [ 2700.945199] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 03:23:56 (1787383436) [ 2708.427812] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 03:24:04 (1787383444) [ 2715.475936] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 03:24:12 (1787383452) [ 2723.601236] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 03:24:19 (1787383459) [ 2732.191365] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 03:24:28 (1787383468) [ 2739.876335] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 03:24:36 (1787383476) [ 2746.573341] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2446 sec ========== 03:24:43 (1787383483) [ 2746.967591] LNet: 139989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2746.971681] LNetError: 139989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2746.981622] LNet: Removed LNI 192.168.202.46@tcp [ 2747.521484] Key type .llcrypt unregistered [ 2747.523087] Key type ._llcrypt unregistered [ 2759.331724] Key type ._llcrypt registered [ 2759.338591] Key type .llcrypt registered [ 2759.431336] LNet: 140463:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2759.431336] mask usage: [+|-] ... [ 2759.669500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2759.686655] alg: No test for adler32 (adler32-zlib) [ 2760.784818] Lustre: Lustre: Build Version: 2.17.54_115_ge5824fa [ 2761.095526] LNet: Added LNI 192.168.202.46@tcp [8/256/0/180] [ 2762.767188] Key type lgssc registered [ 2763.875218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2825.030817] Lustre: Mounted lustre-client [ 2825.263302] LNet: 141932:0:(debug.c:376:cfs_str2mask()) unknown mask 'neterror'. [ 2825.263302] mask usage: [+|-] ... [ 2830.908977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2839.080760] Lustre: Unmounted lustre-client [ 2875.656824] Key type lgssc unregistered [ 2875.901959] LNet: 142918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.906601] LNetError: 142918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.916476] LNet: Removed LNI 192.168.202.46@tcp [ 2876.513821] Key type .llcrypt unregistered [ 2876.515777] Key type ._llcrypt unregistered